Apache airflow for beginners | Executors Architecture | Comparison

#apacheairflow #airflowforbeginners #apacheairflowarchitecture #datascience #datapipelines

Second video on series Apache airflow for beginners, focusing on the high-level architecture of airflow. We'll be discussing and comparing the architectures of 4 major executors namely Sequential, Local, Celery and Kubernetes Executors.

By the end of this video, you'll know how airflow works under-the-hood and which executor suits for your use-case.

If you're new to airflow, be sure to watch the first part of the series here

Watch next video here:
Run Airflow 2 via Docker -

Run Airflow on Kubernetes

👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.

⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.

[maxcoteclearning]

Run airflow on multinode kubernetes cluster - tutorial out now

[PedroCarneiroJr]

At 5:12 you'll need to reedit the audio because there's an overlapping audio on the top of the main one.

[shaikhmushfikurrahman]

Great explanations! Appreciate your effort and works!

[kkmahapatra]

Brilliant explanation on point . Thanks 👍

[jatingrover]

Thanks for sharing knowledge. It is great video indeed. Can you post some content for single node and multi node concept of Airflow. What all components run on same machine/node and what all run on different machine/node. How different components work in conjunction with each other for Airflow execution .

[NathanPrice-qrwn]

With a Celery Executor could you have the workers behind a HorizontalPodAutoscaler to not always have a fixed number of workers or does something about Airflow prevent that kind of dynamic scaling?