Dynamic Tasks in Airflow

In this webinar, we provide an overview of the dynamic task mapping feature that was first introduced in Airflow 2.3 and expanded further in Airflow 2.4. Previously, tasks could only be generated dynamically at the time the DAG was parsed. Now, with dynamic task mapping, you can create tasks at runtime based on external criteria, without knowing in advance how many tasks you need.

The webinar, which reviews the basic functions of dynamic task mapping through pertinent use cases, answers questions like:

-What is the difference between static and dynamic tasks in Airflow?
-How do you implement dynamic task mapping?
-What are the benefits of dynamic task mapping?
-What is a dynamic DAG and how does it compare to a dynamic task?

[pawelwiszniewski]

Great guide, very well explained, thank you!

[shrimp_bucket]

Hi can I get confirmation? Around 11:35 or so you seem to say that only when the DAG is triggered and begins to run, the task mapping occurs? So I cannot retrieve rows in previous tasks and trigger a dynamic task using that information? It has to be available at runtime?

So I should create a Orchestrator DAG that would trigger and input these rows of data into the child DAG to then have immediately available for my Dynamic Tasks?

[dan-takacs]

great video. I'm trying to make this work with LivyOperator do you know if it can be expanded or partial arguments supplied to it?

[VijayGopalParasuram]

Hello, This is very well explained. Thank you! The map index under the mapped tasks are currently numbered from 0. Would it be possible to update the map index to something meaningful ?

[sri]

When you perform the dynamic tasks that could scale in parallel based on your input dataset, what happens to the task log after the dag is finished?

[DanielLee-tkuo]

Thanks for very helpful video. could I get any other guide to apply dynamic tasks on other operators like athenaoperator?

[NK-mycg]

Expand Operator will execute sequentially or parallel?