Implementing Data Quality Checks in Airflow

Watch On Demand

Hosted By

  • Kenten Danas Kenten Danas Lead Developer Advocate
  • Benji Lampel Benji Lampel Ecosystem Engineer

Please note: The webinar stream ended prematurely — about 40 minutes into the webinar, during the Q&A session — due to a technical error. We apologize for the inconvenience.

In this webinar, we discuss the benefits of the Common SQL provider package, a consistent, easy-to-use, and versatile set of operators for implementing data quality checks in your pipelines. In particular, we focus on the SQLColumnCheckOperator and SQLTableCheckOperator, both part of the provider package, and how they can work with OpenLinage.

The webinar shows you how to effectively use SQL for data quality checks, and answers questions like:

  • Why does the Common SQL provider exist and how does it work?
  • How do I implement column-level and table-level checks in my DAGs?
  • How does the Common SQL provider operate with OpenLineage?

All of the sample code shown in this webinar can be found in this repo.

Astronomer Apache Airflow Fundamentals Certification badge

Get Apache Airflow Certified

If you want to learn more about how to get started with Airflow, you can join the thousands of other data engineers who have received the Astronomer Certification for Apache Airflow Fundamentals. This exam assesses an understanding of the basics of the Airflow architecture and the ability to create simple data pipelines for scheduling and monitoring tasks.

Learn More About Certification