Sifflet Provider for Apache Airflow

This package provides operators and hook that integrate Sifflet into Apache Airflow. All classes for this provider package are in the sifflet_provider Python package.

Installation

You can install this package on top of an existing Airflow 2.1+ installation

pip install airflow-provider-sifflet

The package supports the following python versions: 3.7, 3.8, 3.9, 3.10

Configuration

In the Airflow user interface, you can configure a Connection for Sifflet in Admin -> Connections -> Add a new record.

You will need to fill out the following:

Connection Id: sifflet_default
Connection Type: Sifflet
Sifflet Tenant: <your_tenant_name> (for SaaS deployement)
Sifflet Backend URL: <your_backend_url> (for Self-hosted deployment)
Sifflet Token: <your_sifflet_access_token>

<your_sifflet_access_token>: you can find more information on how to generate it here

One of the following is required:

• <your_tenant_name>: if you access to Sifflet with "https://abcdef.siffletdata.com", then your tenant would be abcdef
• <your_backend_url>: full URL to the Sifflet backend on your deployment, for instance: https://sifflet-backend.mycompany.com

Modules

Operators

SiffletDbtIngestOperator

SiffletDbtIngestOperator sends your DBT artifacts to the Sifflet application.

Example usage:

from sifflet_provider.operators.dbt import SiffletDbtIngestOperator

sifflet_dbt_ingest = SiffletDbtIngestOperator(
    task_id="sifflet_dbt_ingest",
    input_folder="<path to dbt project folder>",
    target="prod",
    project_name="<dbt project name>",
)

SiffletRunRuleOperator

SiffletRunRuleOperator Run one or several Sifflet rules - requires rule id(s).

Example usage:

from sifflet_provider.operators.rule import SiffletRunRuleOperator

sifflet_run_rule = SiffletRunRuleOperator(
    task_id="sifflet_run_rule",
    rule_ids=[
        "3e2e2687-cd20-11ec-b38b-06bb20181849",
        "3e19eb3e-cd20-11ec-b38b-06bb20181849",
        "3e1a86f1-cd20-11ec-b38b-06bb20181849",
        "3e2e1fc3-cd20-11ec-b38b-06bb20181849",
    ],
    error_on_rule_fail=True
)