Airflow Dynamic DAGs: The powerful way with Jinja and YAML

Airflow Dynamic DAGs: The powerful way with Jinja and YAML
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow

Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher complexity and subtle tricky things that you must know.

Ready?

Lets go!

[MarcLamberti]

Hey folk 👋
Here is the updated video with the sound fixed: Dynamic DAGs in Apache Airflow for beginners

Enjoy ❤

[litan]

thank you for the great tutorial again. I love the airflow series.

[egehanyorulmaz]

I believe, this method is much more reliable if you are going to generate multiple dags with different schedule intervals. I was using dynamic dag generation with globals()[dag_id]=dag, but if you are working with APIs or clients, scheduler & worker don't work as expected. Also in the documentation, isolated DAG files are suggested, therefore Airflow community must be using your methods as well. Generator.py can be scheduled to generate the files using cron job, and it will be good to go :) Thank you for the content you have shared and informed Airflow users. It was extremely helpful for me :)

[timerelaxingsound]

awesome! that's what i was looking for)

[BrianMoore-bv]

This is informative but I think the piece that I always struggle with is whats the best way to keep the load on the scheduler down if I need to pull data to build the dag based on whats in a database.

[claudee]

Hi! Thanks for your video. Very helpful. Just a comment, you're recording your voice through just one channel, and music in the other, which makes it a bit quieter than normal.

[rajhindocha]

Hi Marc, thank you very much for sharing great information! one quick question though, this could have also done by using normal replace method as well where it will replace the values it receives from yaml in dag template and generate new dag for each yaml config files. Any specific advantage of using jinja2 templates instead?

[tehmoonrulz]

Hi Marc, can you please elaborate on the risks of using config with a loop and globals? I would agree that having one file per DAG is more ideal but you also linearly increase your codebase for each config so that con needs to outweigh the risks/downsides of the loop/global approach

[shiroyashazoro]

Hey. Can I access the airflow instance which is deployed on astronomer using kubernetes?? I mainly want meta data to access

[Fizzility]

Hi Mark, I am new to airflow and I was hoping you’d be able to help provide me guidance on this:
I want to create a job that retrieves
1.api key based on params (ie params=api id)
2. make api call
3. load data
However I need to do this for many different Apikey (ie do this for 100+ different api keys, but function is the same across all, only difference for each job is params of step1) how would you approach this?
Would you dynamically create individual dags based on params for each api key?
Would you have a parent dag to generate subdags for each apikey?
Or any other ideas?

Thanks a lot in advance!

[Arshan]

Hi Marc, first of all, thanks a lot for teaching us airflow! I also got your course on Udemy but I had a question, there you ask us to use virtual box but I didn't and I used your YouTube video to setup airflow using docker in 5 minutes. That shouldn't be a problem for the rest of the course, right?

Edit: fixed tracing to teaching

[gowthamch]

Hmm.. why can I only hear music but not the voice from my right earbud.
great video nonetheless thank you !!

[RajeshSamson]

Do you have any git repo this example?

[nataliaresende]

I never managed to install airflow to start learning…the local host 8080 never opens to me