Kubernetes for Modern Data Engineering: An End to End Data Engineering Project

In this video, we dive deep into the world of Kubernetes, a powerful tool for managing containerized applications, and explore its applications in the field of data engineering.

📝 What You Will Learn:
✅ Setting Up Kubernetes on Docker: We start from the basics, showing you how to set up Kubernetes on Docker. This section is perfect for those new to Kubernetes or looking to refresh their knowledge.
✅ Mastering kubectl: Learn how to effectively use kubectl to manage your Kubernetes cluster. We'll cover essential commands and tips for smooth navigation.
✅ Deploying the Kubernetes Dashboard: Step-by-step guidance on deploying the Kubernetes Dashboard, a user-friendly interface for managing Kubernetes clusters.
✅ Running Apache Airflow with Helm Charts: Discover how to run Apache Airflow, a leading tool for orchestrating complex computational workflows, on Kubernetes using Helm charts.

Timestamps:
0:00 Introduction
1:05 Setting up Docker on your machine
3:42 Managing Resources for Docker and Kubernetes
4:20 Enabling Kubernetes on Docker Desktop
5:47 Setting up Kubectl and other Kubernetes Management Tools
11:06 Setting up Helm charts
13:08 Working with Kubectl commands
17:25 Setting up Kubernetes Dashboard and Managing the cluster
20:56 Generating and managing Kubernetes dashboard Token and secrets
22:54 Kubernetes Service Accounts
23:48 Kubernetes Role Based Access Control (RBAC)
25:20 Kubernetes Secrets and Tokens
29:10 Kubernetes Dashboard
36:38 Setting up Apache Airflow on Kubernetes with Helm Charts
40:13 Accessing Apache Airflow on Kubernetes
43:08 Reconfiguring Apache Airflow on Kubernetes
47:40 Connecting Apache Airflow DAGs to Kubernetes
1:01:19 Working with Multiple DAGs on Kubernetes
1:18:41 Optimising Airflow DAGs on Kubernetes
1:23:50 Outro

Resources 🔗:

Tags ✨:
Kubernetes, Data Engineering, Apache Airflow, Docker, Helm Charts, Kubernetes Tutorial, Data Processing, Cloud Computing, DevOps, Technology Education, Kubernetes Dashboard, kubectl, Containerization, Workflow Automation, Big Data, Tech Tutorial, Kubernetes Cluster Management, Kubernetes for Beginners, Scalable Data Engineering, IT Infrastructure, Cloud Services, Kubernetes in Data Science, Kubernetes and Airflow, Continuous Integration, Continuous Deployment

Hashtags ✨:
#Kubernetes #DataEngineering #ApacheAirflow #Docker #HelmCharts #DevOps #Tutorial #TechEducation #KubernetesTutorial #DataProcessing #CloudComputing #TechnologyTutorial #Learning #educational

[CodeWithYu]

Guys, please don't forget to LIKE, SUBSCRIBE and SHARE! 🙏🏻🥺

[kunleade-oni]

Good way to learn after learning the basics of Kubernetes.

[tarik_el_oukili]

thanks Yusuf, a very informative video as usual

[EhsanIrshad]

you are legend to grab attention from your thumbnil.. you are awesome researcher thank you for this video man.. hats offff

[eduardosteps]

Great video, Yu. Thank you so much. I'm looking forward to seeing the next part, where you teach us how to use a local DAG on our machine and keep it synchronized in Kubernetes, just like we do in Docker with persistent volumes.

[vanessaanna-mariaandersson]

Thank you so much for sharing this! I followed your steps and completed the project after much troubleshooting and help from ChatGPT. I appreciate your time and effort in making this video. I'd like to give you some feedback for next time: Add the seniority level someone should have in your description before watching, as I struggled to understand many of your steps and code. I am a junior in the era and would appreciate a more detailed description and analysis of each step. Thank you so much again! You are doing a fantastic job!

[jeffphan]

thank you so much for your videos!!! looking forward to seeing your next part of this. I'm about to apply this video to your Realtime data streaming

[spider]

Very nice Yusuf, please can you do a series of project with GCP

[vincevdam]

Thank you Yu. Been watching your videos hoping to transition into a data engineering role in the future.

[howdyservices]

Excellent tutorial. Appreciate your effort in explaining everything in great detail. I believe spark-on-k8s will be a great addition to this video. Thanks for the content.

[wiss]

Thank you youssef for your efforts you put in your videos...I am working in snowflake projects and I am waiting for DBT and Snowflake end to end projects...Thank you!

[adityanjsg]

I have used django in K8s, Airflow cloud compose, Azure datafactory. This is new and helpful too!

[PrakashReddyK]

Dude, first of all thank you very much for all your efforts and explanations, you are awesome, looks like you are renting in my head, i just completed setting up kuberntes on my 4 node raspberry pi 😅 and was exactly looking for this kind of scenario to setup an application on the k3s, thank you for posting this video, one small request is, can you please make a video of installing spark operator on kuberntes cluster and submit a job, please, pretty please, thanks in advance

[RafaVeraDataEng]

Hi! I have already this project in Kubernetes running and in the same project I added the dbt-Bigquery project.
both are running and working as expected, even the dbt-bigquery inside a docker container.

My goal is lauching the dbt process from a DAG using KubernetesPodOperator.. but it fails every time.
I'm still not seeing the logs in Airflow UI... how have you solved this issues? I tried to set get_logs=True, is_delete_operator_pod=False without success...

any suggestion please?

[DhruvanGowda-vopz]

How to implement this in cloud platform, can you help me on this..?

[atom]

Need a video on running spark cluster on K8s from you

[satwikkumar-eqfm]

Hi @CodeWithYu I'm unable to access the airflow as the credentials are incorrect. Could you please help me?

[ashfaqurprotick]

i am the first viewer in 28 seconds . Waiting for your videos.

[kashifsadiq]

sir start dataops course with projects

[shantelcehas]

I find these type of videos useful, but I feel you rush some of the core concepts, for example how do I get to the container section. You don't really explain that for a beginner friendly tutorial. It would be nice to have a complete beginner session. I