How to Orchestrate Machine Learning Workflows with Airflow

Watch On Demand

Hosted By

  • Jeff Fletcher Jeff Fletcher Director of Field Engineering EMEA
  • Kenten Danas Kenten Danas Lead Developer Advocate

This webinar provides an overview of machine learning orchestration, its importance in managing ML workflows, and how it fits in with MLOps. We cover everything you need to know to use Airflow to streamline your ML workflows and MLOps environments including orchestrating tasks such as data ingestion, data processing, model training, model deployment, and ongoing model monitoring. Questions covered in this session include:

  • What is machine learning orchestration and how can Airflow be used for this purpose?
  • What operators are available for Airflow that can be used for machine learning orchestration?
  • What are some best practices for setting up and managing an Airflow environment for ML workflows?
  • How can I optimize Airflow performance for ML use cases and avoid common pitfalls?

All code covered in this webinar can be found in this repo.

Astronomer Apache Airflow Fundamentals Certification badge

Get Apache Airflow Certified

If you want to learn more about how to get started with Airflow, you can join the thousands of other data engineers who have received the Astronomer Certification for Apache Airflow Fundamentals. This exam assesses an understanding of the basics of the Airflow architecture and the ability to create simple data pipelines for scheduling and monitoring tasks.

Learn More About Certification