What's new in Apache Airflow 2.10?

🚨 Join me at the Airflow Summit between Sept 10 and Sept 12, 2024, in San Francisco. Use the code 20DISC_MARC for a 20% discount

Here are the significant updates:

✅ Dynamic Dataset definition
💻 Create datasets dynamically from tasks! No more static datasets

✅ Big update on the Datasets view
💻 Better track which dataset triggers which DAG runs and more!

✅ Hybrid Executors!
💻 Configure multiple executors to efficiently run your tasks

✅ Dark mode
💻 That's right! Save your eyes (still experimental, though)

✅ New button to re-parse DAGs
💻 Stop wasting time, hit the button to get your DAG updates right away

✅ New concat() method
💻 Combine multiple outputs from upstream tasks

And more!

👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow

[kamranalisyed]

Wow love it. Especially multi executors.

[LoveDarkChocolate]

I would like to ask you about the DAG being scheduled on a Dataset or Dataset Alias. I assume that such DAG will only run if those datasets are modified by another DAG, not by any event that modifies those files outside the Airflow architecture?

[rembautimes]

Just joined your Channel. Very good content

[surawutjirasaktavee]

Thank you for sharing!
Can you have the course that explains how to set a standalone dag-processor for multi-tenant teams using the official helm chart?

[monkshandilya]

Trying to run airflow scheduler on both ec2 nodes in a Load Balancer setup .but it's not working as expected, any suggestions?

[Empusas]

I like your content and have your courses in Udemy. What I am searching for, which is not included in your courses, is how to properly size airflow in a docker environment. If I need more workers, is it better to increase the celery workers or add new worker containers. What are the limiting factors of the host and how to monitor them. I see that my host, which is a vm, is mostly idle, but I have only 16 parallel celery workers.