executor out of memory spark | spark memory management | Lec-16

In this video I have talked about spark memory management. I have shown how executor OOM occurs in spark. If you want to optimize your process in Spark then you should have a solid understanding of this concept.

For more queries reach out to me on my below social media handle.

My Gear:-

My PC Components:-

[prashantmehta]

No.. Words... .. Thanks to the Universe and you(sir) I found this channel..

[vishaljare]

In-depth explanation .even in paid courses they are not teaching this kind of in-depth explanation.hats off bro love from Maharastara❤

[vinayak]

Itna clarity se kisine nahi bataya bhai, bahut bahut dhanyawad ❤

[anuragtiwari]

Excellent stuff manish.. 👍👍 I Was eagerly waiting for this video..

[lkmlujv]

Hello manish first of all thank you for simplifying the spark for us it helps a lot. If possible Please make a one video on cache and persist.

[gkgk]

he buddy really I'm appreciate you and thanks for this playlist

[ranjithrampally]

I thank you from bottom of my heart, this series is really amazing. I request you to create a video on list of common issues/erros faced in pyspark and how to resolve them.

[radheshyama]

Please is series ko continue rakhiyega

[mmohammedsadiq]

Thanks for your extensive preparation and the explanation, If u can share these board writeups as a PDF or ppt, will be so useful for the interview revisions

[TheSmartTrendTrader]

Hey. Thanks for the good insights. How are you able to write texts through the cursor in one note for windows 10? When I am trying to do it displays texts as jumbled varying size as how I move my cursor.

[lpqejjm]

Can you please upload the video of map and flatmap transformation and difference and another one is share your project challenges and how you optimized the transformation in realtime for that project

[abinashpandit]

2 din m apka pura spark theoretical dekha. Spark pura dimag m gush gaya hai. Loved your teaching style. Ita depth m kahi Maine nahi padha na samjha. You are the Khan Sir of the Data World ❤

[akash]

Bhai ab jaldi videos lane ka try hone wala h ache se explain kia hi lge rho...full support h. ❤

[TechnoSparkBigData]

All these things applicable to databricks as well?

[khushbooagarwal]

If data spills on the deisk, this disk is available in executor ot at common space like off heap memory?

[ankandatta]

Bhai, AQE solve karega na partition skew ko?

[sanketraut]

Hi Manish sir, I'm getting out of memory error: Java heap space. But while submitting/writing dataframe in hdfs. I have partitioned data then I have applied bucketing on it. Then I coalesce it as optimization. And while writing it into hdfs error occurred. I have increased driver memory, memory over head but still problem is same?

[worldthroughmyvisor]

what is meant by spilling to disk ?
do you mean a storage device like SSD or HDD ?
or do you mean spill it in the RAM itself ?

[rhhrccr]

not sure ... but when 10 gb is allocated to executor memory, then 1 gb(10%) of allocated memory will get by executor overhead memory. so 9gb for executor and 1gb for executor overhead
unlike 11 gb explained in video... correct me if i wrong ?
get this answer in gemini

[TheBest-yhyj]

Hi, Can i get a pdf or image of the flow, as ss lene k baad kuch samjh nahi raha hai