Spark Scenario Interview Question | Persistence Vs Broadcast

#Spark #Persist #Broadcast #Performance #Optimization
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more

About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.

Visit us :
Twitter :

Thanks for watching
Please Subscribe!!! Like, share and comment!!!!

[mrkrish]

This scenario was not clear when i go through external video but after your explanation i understood the difference. excellent

[sumitgandhi]

Thanks for explaining it in easy way :)

[umamahesh]

Data in memory and data in disk will not occupy same space. So if it is 12gb on disk in memory it can be 18gb

[gemini_]

Is it approximately 12GB after persisting? Any significant overhead when the data is in memory?

[HemanthKumar-cmlv]

Hi viresh, Thanks for the video,
can you confirm the below statement,

In Persist executors save partition of the data frame in memory and in broadcast executor will save the entire data frame in memory?

[hiteshpatil]

why here u r taking only 3 no. of executor?

[gemini_]

In the case of broadcast, why do we have to include the 12GB of existing DF? I feel it is unfair to compare persist with broadcast. It is possible to avoid the 12GB?

[mateen]

Hi Viresh, Thanks for this nice video....I believe that the broadcast variable is used to broadcast a small table and join it with a huge table which would avoid shuffling....What happens if we broadcast a table with more number of columns into the executors? Assuming that the broadcast table is larger in size because of having more number of columns.

[xxx]

I dont understand the premise of sending the whole dataset to each executor. you are defeating the purpose of spark which is distributing data over network.
Second thing is, if you clearly state what comparison is then this is really a straightforward task (I guess you forgot about garbage collection of original 12 Gb Data as well, correct me if I am wrong). I would be more interested in comparing data in transit comparison.
Lasty I think more challenging compare shuffle and broadcast operation.

[mahendramaurya]

why the number of partitions are three in case of broadcast join, Can we keep it low like 1or 2 orr why not to keep all the partitions into single executor

[siddhantpathak]

Hi Viresh, I am new to Spark so cut me some slack for asking newbie questions.

In persistence, the data frame is held either in memory or disc. Suppose the data from the data lake was held in executor memory which is in the given case is 4 GB and it is completely occupied.
Now I want to read another data frame, then how executor will deal with it since its memory is already occupied by the previous persisted data frame.

In Broadcast, the memory footprint is said to be 4 times the data frame memory. Where does this memory come from since each executor got only 4 GB? And I also read somewhere after Garbage Collection it is only left 3 times. Why is it so?

In persistence, the data is said to be stored in memory. Is it just the executor memory which is 4 GB or entire system memory?

Thanks in Advance.

[aneksingh]

Hey Viresh, from where do you study these concepts . please share resources

[anantababa]

not clear ..you have explain about the persistence ..

[suryasatish]

broadcast variable is copy per node right ? why it will 36 GB ?