Spark [Executor & Driver] Memory Calculation

#spark #bigdata #apachespark #hadoop #sparkmemoryconfig #executormemory #drivermemory #sparkcores #sparkexecutors #sparkmemory

Video Playlist
-----------------------

YouTube channel link

Website
Technology in Tamil & English

#bigdata #hadoop #spark #apachehadoop #whatisbigdata #bigdataintroduction #bigdataonline #bigdataintamil #bigdatatamil #hadoop #hadoopframework #hive #hbase #sqoop #mapreduce #hdfs #hadoopecosystem #apachespark

[sangramrajpujari]

This is a real A to Z calculation for memory. Thanks for the useful video.

[neelbanerjee]

Sometime question from interviewer that what is the data size of your project and how you do this memory allocation based on data size? Could you please make a video to explain those real cases depend upon data size

[pavanpavan]

This is the best video I have watched on Executor Memory calculation. Thank you brother.

[svJayaram]

Thank you so much. It's very clear to me

[gouthamanush]

What if your input size keeps changing? On one day if it's 1GB and another day it's 1TB, would you still suggest the same configuration? Can there be a correct configuration in such cases?

[ananthb]

Hi anna, Vanakkam. My doubt is... How can this calculation be suitable for all the jobs.. Taking the same cluster configuration which you explained in the video, for all the jobs even though the size of data handling will differ from job to job, still we will calculate according to whole cluster configuration...please explain I am really confused.

[vikaschavan]

Do we need to consider existing running jobs in prod environment while giving these parameter values for our spark application. Thanks in advance.

[sachinchandanshiv]

Hi Sir,

Can you please explain, What is the practical Hadoop cluster size in projects of companies?

[bhavaniv]

Thanks for the detailed explanations 👍

[sukanyanarayanan]

Hi, One clarification
In real time scenario, we need to decide the resource based on the file size we are going to process.
Can you please explain how to explan how to determine the resource based. On the file size

[macklonfernandes]

Thanks for info.
So i have 1 master and 1 worker with 4cpu and 16gb and available memory is 12gb
So when i submit spark job on yarn with driver and exector memory 10gb and core as 4
Its not able to assign the passed values.
Inturn 1 core and 5 or 8 gb is assigned for executor
Any help would be helpful

[vijjukumar]

If my spark reads data from event hub what is the recommended partitions count at Event hub. if partitions count is 10 only one driver connect to all partitions and sends to worker nods?

[sreelakshmang]

My core node in emr has 32 gb memory and 4 cores, but when checking spark ui, i can see only 10.8gb and 1 core being used. why is that?

[svdfxd]

How do the dataframe partitions impact the job?

[nithinprasenan]

What if... I have a stand alone mode... And I have 16core and 64gb of ram.. how to calculate executor and driver memory

[WritingWithShreya]

Hi, How minimum memory is 5GB? Please explain?

[AshokKumar]

Could you explain When do I increase number of executors and when do I increase no of cores for a job?

[dhnushl]

you have data engineering course notes pdf

[vivekrajput]

Still a question for which I am not able to get proper answer.
Suppose you have 10 gb data to process, using data volume scenario, please explain number of executor, executor memory, driver memory.

[guptaashok]

Thanks for this nice video, have a question.
suppose i have worker 2 nodes each having 4 core and 14GB memry

scenario 1:
by defualt databricks creats 1 executor per node, it means each executor will have 4 cores and 14 GB, hence can run 4 parallel tasks

total 8 parallel tasks.

Scenario 2:

If i configure databricks to have 1 executor per core and configure 3 GB memory per executor

I can have 8 executor in total, which means 8 task can run in parallel, each will have 3 gb

both ways i can run max 8 tasks in parallel, on what basis i should choose my distribution model to get optimal perforance?