Workshop (Workshops - Day 1) - Airflow 2.0 for ML pipelines – design, implementation and management

Abstract: With a lot of changes under the hood with Airflow 2.0, the workshop aims to give an overview on major updates in Airflow 2.0 from 1.0, major components and working of Airflow and hands-on demo of implementation and management of an end-to-end Machine Learning pipeline. Without a pipeline in-place, management of multiple Machine Learning stages in production can be difficult. This gives an overview of simplified process and management of Python based ML projects using Airflow.

05 min: Introduction
05 min: Major changes in Airflow 2.0
05 min: Pre-requisites setup overview
10 min: Walkthrough of different backend components
10 min: Different stages of a DAG file – steps and operators
10 min: Dynamic DAG creation to improve parallelism
15 min: How to trigger Airflow DAG runs
15 min: Debug and clear Airflow task errors
10 min: Overview of production-level Airflow-based architecture
05 min: Wrap up questions

Speaker: Alen Jacob

[shrikanthsingh]

Hello PyCon, It was a great tutorial to start with. However I am trying to improvise by adding plots to the mlflow artifacts. To accomplish this I want to add seaborn and matplotlib. I tried adding them to requirement.txt and also to environment.yml and running the compose again. But the airflow UI still reports the error "No module named seaborn". Any help is appreciated