Spark Performance Tuning | EXECUTOR Tuning | Interview Question

#Spark #Persist #Broadcast #Performance #Optimization
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more

About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.

Visit us :
Twitter :

Thanks for watching
Please Subscribe!!! Like, share and comment!!!!

[RohanKumar-mhpt]

Very Nice and clear explanation before this video i was very confused regarding executor tuning part now after this video it is now crystal clear.

[TheFaso]

Dude. I feel like I knew nothing about spark in particular before I got my hands dirty with your performance improvement solutions.
Appreciate a lot, got my subscription. Cheers from Germany !

[fahad_ishaqwala]

Excellent videos brother. Much Appreciated. Can you do a video on Performance Tuning for Spark Structured Streaming jobs as well.

[sankarn]

Nice Explanation!!
can we use this approach for tuning/triggering multiple jobs in cluster ??

[aneksingh]

As always best !!! Please include some real simulation example s

[giyama]

This calculation is for just one job, what would be the calculation for multiple jobs running simultaneously?

And how to calculate based on the volumetry?

(Great job btw, tks!)

[mdmoniruzzaman]

Hi, 10 nodes means including the master node?

i have a configuration like this:
"Instances": {
"InstanceGroups": [
{
"Name": "Master nodes",
"Market": "SPOT",
"InstanceRole": "MASTER",
"InstanceType": "m5.4xlarge",
"InstanceCount": 1
},
{
"Name": "Worker nodes",
"Market": "SPOT",
"InstanceRole": "CORE",
"InstanceType": "m5.4xlarge",
"InstanceCount": 9
}
],
false,
"TerminationProtected": false
},

[SpiritOfIndiaaa]

thanks bro, really wonderful explanation.... bro, can you make some vid on how to analyze Stages, Physical Plans etc on SparkUI ...based on that how to fix the issues regarding optimization ... its always confusing a lot to interpret these sql explain plans?

[Dipanki-ck]

What if I have multiple spark jobs in parallel in on spark session

[sivavulli]

Hi Sir, thank you for your nice explanation but if only one job is running over the cluster, that is more meaningful and understandable ..what if there are so many jobs running on the same cluster ??

[whatever-genuine]

How to allocate executers, core and memory if there are multiple jobs running on the cluster?

[KNOW-HOW-HUB]

To process 1TB data what could be the best approach we have to follow

[inferno]

@5:10 can you explain how 20GB + 7% of 20GB is 23GB and not 21.4GB ?

[DilipDiwakarAricent]

If not configure, so what will be the default number choose by spark.

[umeshkatighar]

What If each node has only 8cores?? How does spark allocate 5cores per jvm ?

[manisekhar]

According to your eg. How much GB if data can be processed by spark job??

[snehakavinkar]

How to decide these configurations for a certain volume of data? Thank you.

[snehakavinkar]

Is there any upper or lower limit to the amount of memory per executor?

[rikuntri]

One executor is having four core so it can handle one task or 4 at a time

[girijapanda]

7% of 21GB = 1.4 GB am I missing something here