Apache Airflow with Spark, Pyspark, Java, Scala for Data Engineers || Full Course

In this course, you will create an end to end data engineering project with the combination of Apache Airflow, Docker, Spark Clusters, Scala, Python and Java. You will create basic jobs with multiple programming language, submit them to the spark cluster for processing and see live results.

⏳ Timestamps:
00:00 Introduction
00:57 Creating The Spark Cluster and Airflow on Docker
11:00 Creating Spark Job with Python
28:51 Creating Spark Job with Scala
37:37 Building and Compiling Scala Jobs
43:23 Creating Spark Job with Java
58:51 Building and Compiling Java Jobs
1:06:15 Cluster computation results

✅ Don't forget to LIKE, COMMENT, SHARE and SUBSCRIBE to our channel for more data engineering projects.

🔗 Resource Links:

📢 Stay connected:

🏷️ HashTags:
#ApacheAirflowCourse #DataEngineeringWithAirflow #AirflowOnDocker #SparkDataProcessing #ScalaForSpark #JavaDataEngineering #MavenProjects #BigDataAnalytics #WorkflowAutomation #FullCourse #FreeCourse #Educational #dataengineering

👍 If you found this course helpful, please LIKE and SHARE the video, and leave your thoughts in the COMMENTS below.

🔔 For more tutorials and complete courses, make sure to SUBSCRIBE to our channel and hit the bell icon for notifications!

[CodeWithYu]

Thanks for watching! Kindly Like and Subscribe for wider reach 🥺

[jordansabo]

This is one of the most underrated coding channels on YouTube. I like how you practically describe every line of what you are doing without getting too in depth. Keep making these!!

[dswithanand]

please make a detail playlist with apache kafka also. Will really appreciate it. your contents are really good and very practical.
Thanks in advance.

[วชิรพันธ์สุรศร]

Thanks for your video! By the way, do you know if it works with the Airflow Celery executor?

[selene]

Hi, why are you running the code in a virtual environment? Do i still have to run it in a virtual environment even if I'm using linux? Please reply.

[hemantsah]

Hey I have a similar docker-compose file. When I submit a job using SparkSubmitOperator, it says JAVA_HOME not set. But it is actually there in spark master and worker container i.e. /opt/bitnami/java.
Do you know what might be happening here? Any help appreciated

[muhammadfayyaz]

Looks like the spark-submit operator has some bugs while in its working. I have tried different connection names along with the IP of spark master but still getting the same error.

Cannot execute: spark-submit --master spark-master:7077 --name arrow-spark jobs/python/wordcountjob.py. Error code is: 1.

[charlesokekearu]

Lovely video Mr Yusuf.
I ran into an issue while trying to open my airflow webserver. It didn't come up using localhost8080. What do you think could be the issue? Thank you.

[tungduong]

Thanks for your content. Can I ask you some question? If I want to deploy a Spark Cluster in production so that user can create and submit job base on my job template, can I submit job like you do in this video or I should dockerize the spark job and run it on Spark K8s cluster. Thanks a lot

[swamynaidulenka]

You are the most underrated..
Great content.. 🎉🎉
Hope you will get all the credits.. Soon❤❤❤

[ataimebenson]

Very lovely video, The combination of python job, scala job and java job is very nice

Thank you

[madiounimohsen]

nice video it was very instuctive could u add some videos in which u explain how to dockerize spark app and add some unit tests

[NjunwaWamavoko]

@CodeWithYou quality is not good only 360p

[aliel-azzaouy]

thank you sur, Could you please reupload it in better quality? It would be much helpful! Thanks🙏

[jaswanth]

Hi Yusuf,

Could you please share resources to master Docker and Docker Compose file creation?

[youssefouleddehou]

Thanks a lot bro, Could you share with us a course (docker) for implementing a advanced dockerfiles

[rikinpatel]

Seems like a really informative video! Could you please reupload it in better quality? It would be much helpful! Thanks🙏