deployment mode in spark | edge node in spark | Lec-18

In this video I have talked about deployment mode in saprk. Also I have talked about edge node machines. You should have a solid understanding of spark architecture to understand the concept.

For more queries reach out to me on my below social media handle.

My Gear:-

My PC Components:-

[dayanandab.n]

Manish bhai you are my Dhronacharya of Data Engineering..🙏🙏🙏

[vinayak]

Databricks cluster aur spark cluster ke different samjha do pls🙏🙏🙏

[amankumarsrivastava]

Cluster mode aur Client mode, dono ke case me Application master worker node pe hota hai ? + Aur kaunsi major topics aaengi isme ? Mera interview aarha so mai soch raha alag se padh lu

[pramod]

Thanks Manish for the video wanted to know 2 things 1. where the logs will be generated in cluster mode because if we have to debug how would it be possible to gather logs of all worker nodes 2. if we have to write df as file on a path how can we achieve it in cluster mode because at run time we dont know where(on which node) exactly will be file written by driver

[ruthvikjaini]

How are libraries handled in client mode and cluster mode?

[kwuuvwi]

hello, i guess u made a mistake 11:08 . In cluster deployment mode, executor will not get replicated in multiple worker nodes
@manishkumar please take a look at this

[anud]

In cluster mode, why spark driver is created in worker node? As per spark architecture, spark driver is created in master node wheras worker nodes are meant for spark executors. The master slave concept. Please help me understand 🙏.

[isharkpraveen]

Mujhe abhibi doubt he.... Driver and executor memory kis hisaab se set kiya jaatha he?

[sankuM]

@manish_kumar_1 thank you yeh wala detail me samjhane ke liye!
Ek doubt hain: Cluster mode me driver worker node par kyun rakha jaata hain woh nhi samjha thoda..!!
Like ek proper cluster hain to ek alag se driver node hoga na, same configuration wale nodes ke pool me se driver & executor lene ka logic digest nhi hua.

[mission_possible]

Thanks for the content. Can you please make video on Dynamic partition pruning?

[dineshughade]

Could you provide the diagram which you have drawn?

[gouravkumarkundu]

Hi bhaiya,
I am currently working in master data management(MDM) domain in cognizant...
I want to switch into data engineering...
Can you please tell me how can I do that?
Please guide me🙏🙏

[lalitpatel]

am looking to make a switch.


Currently I am working in cloud support. Most my work is of monitoring and nothing more. I don't think next one year they goanna give me some real experience on technology. Even I don't have access to cloud. It's just a monitoring job. So, currently looking to switch.

I am confused in which field I should choose.

SDE or DEVOPS or DATA SCIENCE or DATA ENGINEER

Don't know really.

[bongworldalways]

Jar file create krna nh dekhaia or one by one step kar late to samaj ne me accha hota not get it