Mastering Apache Spark - Course Details and Overview

#apachespark #course #machinelearning #dataengineering

In this course we will learning Apache Spark from basics to advanced. This is going to be a completely hands on course as well as cover real world data challenges

Course Covers

Spark for Data Engineering and Analysis
Spark for Machine Learning
Spark with H2O AutoML
MLFlow for model tracking and deployment
Spark on GPU
Spark Streaming (Coming Soon)
End to End Data Science Flow
Real World Data Collection
Data Architecture

[pratiksingh]

It is good to see a complete Spark Series, Starting to view it with lots of hope :)

[ijeffking]

Earnestly looking forward to this course. Thank you so much.

[seemunyum]

Thank you so much sir! So excited for this series! Doing this before my interview in a week!

[arvindkumar-ugzf]

Outstanding contents sir thank you very much, sir

[pratiksingh]

Question : In External Shuffle Service, where does the metadata live ?
Somewhere I read :
"When enabled, the service is created on a worker node and every time when it exists there, newly created executor registers to it. During the registration process, the executor informs the service about the place on disk where are stored the files it creates. Thanks to this information the external shuffle service daemon is able to return these files to other executors during retrieval process."
So can we say that enabling dynamic resource allocation (and ESS) has a trade-off that the shuffle data will be written to disk which in case of Static Allocation would have been written in Executor Memory ?
Thanks in advance @AIEngineering

[rushirajparmar]

Hi Srivatsan, I just started with the playlist and its very informative! Thank you so much, because of you many people might be able to understand something about "MLOps". However, I wanted to ask that is such a pipeline feasible for Computer Vision applications? Could you please shed some lights on this?? Thank You!

[ganeshnayak]

Thanks Sir for Awesome Content...can you share your knowledge on DWH and ETL design as well.

[jnana]

Hi Srivatsan. Very informative and useful playlist on Spark..I want to ask you about the challenges or issues one comes across frequently while doing Machine learning or data engineering using Spark/Pyspark.Thanks

[phanisrikrishna]

Hello Sir, Thats a nice course covering pyspark in data bricks. Could you also share the slides and notebooks used in the sessions. Notebook would be helpful for practice and slides for reference. Thank you.

[shubh]

H Sir, are there any videos for spark real time model deployment in this series?

[m.sameerkhan]

Please make a complete tutorial on big data series with coding assignments if possible.

[nandakishorenaidu]

Hi Srinivas, I dont have any Java Background and I have just started with Python.Can I learn Spark with out any Java background.

[sunnyraut]

Sir can you give me the pre requisite for learning spark python and big data . I am 1 year experienced from DWH background with knowledge in SQL and reporting.

[sandeepgupta]

Do I need to follow a specific path to complete the 29 videos. Any specific order ?

[gauravlotekar]

how updated is the course sir? and we using Scala for the course?

[ajayprajapati]

Hello Sir, how can we process streaming data on colab.

[rajvashisthsharma]

Hello Sir, Please help me, I have complete setup for pyspark but while start using SparkContext getting below error : " Py4JJavaError: An error occurred while calling
: Could not initialize class

[nimjae]

Do I need to know hadoop If I want to be data engineer??? or Is it ok to just learn only spark?

[rajiyengar]

Is there a GitHub link for the above? Thanks.

[pratiksingh]

Can someone help me understand that wasnt this the actual difference between MapReduce & Spark Framework . now if Spark is using Map & reduce phase in backend there where is this difference ? @AIEngineering