Spark Runtime Architecture (Cluster Mode) | #pyspark | #databricks

#pyspark #azuredataengineer #databricks

Use the below link to enroll for our free materials and other course.

You can talk to me directly on Topmate by using the below link:

Follow me on LinkedIn
-----------------------------------------------------------------------------
Clever Studies Official WhatsApp Group joining link:
--------------------------------------------------
Follow this link to join 'Clever Studies' official telegram channel:
--------------------------------------------------

PySpark by Naresh playlist:
--------------------------------------------------
Realtime Interview playlist:
--------------------------------------------------
Apache Spark playlist:
--------------------------------------------------
PySpark playlist:

Hello Viewers,

We ‘Clever Studies’ YouTube Channel formed by group of experienced software professionals to fill the gap in the industry by providing free content on software tutorials, mock interviews, study materials, interview tips, knowledge sharing by Real-time working professionals and many more to help the freshers, working professionals, software aspirants to get a job.

If you like our videos, please do subscribe and share within your circle.

Thank you!

[_Sujoy_Das]

Needed this one badly... Thanks Naresh

[PavanKumar-vihd]

Hi Naresh
your way of explanation is excellent.
this is first time i understand spark architeecture is very easy way in Cluster Mode

[pankajchikhalwale]

Please make a short video on the relationship between stages, node, executor, dataframe/dataset/RDD, and core, partition, and task. Want to know what consists what ? And what contains what.

[sagararora-ex]

How cluster manager can create any of the nearest worker as application master because the configuration of the master can be different. So will it not go and create master to the machine that is configured for master role. With memory allocated to master depending on the type of task.

[TechGuyTelugu]

What about the node manager do in this architecture

[sohaibshah]

In your Spark Architecture the driver is created on worker node while in other Architecture I can see the driver on Master Node Why? Thanks

[sohaibshah]

what will happen if a worker node goes down where we have created driver?

[vivekmerugu]

Hi Naresh,

Thank you so much,

This help me a lot. ❤

Naresh, I've execute my spark application in cluster mode (yarn) in emr cluster,

My spark application is failing with an exception saying application master container failed 2 times, exists with 137 code.

This exceptional is occuring for only one dataset which I'm processing with spark application.

For other datasets, my spark application is working fine.

The dataset for which spark application is failing having large input payload, ( one record with 25000+ characters ).

I tried increasing the driver memory and executor memory, now this time, I'm getting an exception while deserialization of input payload.



Any suggestions how to resolve this issue.

It will be helpful, please