Airflow BigQuery Reservation Provider

Warning This package is a pre-released of the official apache-airflow-providers-google package. All of these operators will be integrated to the official package, soon.

This repository provides an Apache Airflow provider based on BigQuery Reservation API.

Airflow Operators

• BigQueryReservationCreateOperator: Buy BigQuery slots (commitments) and assign them to a GCP project (reserve and assign).
• BigQueryReservationDeleteOperator: Delete BigQuery commitments and remove associated ressources (rservation and assignment).
• BigQueryBiEngineReservationCreateOperator: Create or Update a BI engine reservation.
• BigQueryBiEngineReservationDeleteOperator: Delete or Update a BI engine reservation.

You could find DAG samples here.

Requirements

• A Google Cloud connection has to be defined. By default, all hooks and operators use google_cloud_default.
• This connection requires the following roles on the Google Cloud project(s) used in these operators:
  • BigQuery Resource Admin
  • BigQuery Job User - Required for BigQueryReservationCreateOperator because of the reservation attachment check.

Defining a new dedicated connection and custom GCP role could be good practices to respect the principle of least privilege.

How to install

pip install --user airflow-provider-bigquery-reservation