35. Databricks & Spark: Interview Question - Shuffle Partition

Azure Databricks Learning:
==================

Interview Question: What is shuffle Partition (shuffle parameter) in Spark development?

This video covers more details about shuffle paramater

#DatabricksInterviewQuestion #DatabricksInterview #SparkInterviewQuestion #SparkInterview #PysparkInterviewQuestion #PysparkInterview #BigdataInterviewQuestion #BigdataInterviewQuestion #BigDataInterview #SparkShuffle #SparkShufflePartition #SparkShuffleParameter #Databrickshuffle #DatabricksShufflePartition #DatabricksShuffleParameter #PySparkShuffle #PySparkShufflePartition #PySparkShuffleParameter #SparkTransformation #SparkWideTransformation #SparkPerformanceTuning #SparkPerformanceOptimization #SparkPerformance #SparkOptimization #SparkTuning #DatabricksTransformation #DatabricksWideTransformation #DatabricksPerformanceTuning #DatabricksPerformanceOptimization #DatabricksPerformance #DatabricksOptimization #DatabricksTuning #PysparkTransformation #PysparkWideTransformation #PysparkPerformanceTuning #PysparkPerformanceOptimization #PysparkPerformance #PysparkOptimization #PysparkTuning #DatabricksTutorial, #AzureDatabricks #Databricks #Pyspark #Spark #AzureDatabricks #AzureADF #Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial databricks spark tutorial databricks tutorial databricks azure databricks notebook tutorial databricks delta lake databricks azure tutorial, Databricks Tutorial for beginners, azure Databricks tutorial databricks tutorial, databricks community edition, databricks community edition cluster creation, databricks community edition tutorial databricks community edition pyspark databricks community edition cluster databricks pyspark tutorial databricks community edition tutorial databricks spark certification databricks cli databricks tutorial for beginners databricks interview questions databricks azure

[PavanKumar-ttmm]

Expecting some More concepts on Pyspark Raja. Good Effort

[antonyvinothans]

05:18, so which means cores and partitions, both are same ?

[ranjansrivastava]

Raja pls make clear here:- The default number of partitions for the RDD/Dataset is 8 and the default partition size is 128 MB On the other hand the default partition for the shuffling partition is 200 and size is 128 MB as well. It means shuffling partition is applied on the worker node and RDD/Dataset partitions will be implemented on the driver node. Please share your inputs on this.

[abhaybisht]

Great content Raja. Make one detailed video on Spark performance and optimizations.

[omprakashreddy]

So, If I understand correctly after you set some value for shuffle partitions and later after shuffling if you don't get expected performance then we go with "repartition or coalesce " right?

[aayushisaxena]

I have 500 gb output dataframe with no aggregate or joins, and it needs to be written to a table, will repartition or shuffle operations improve parallelism?

[BlingKing]

Why there will be disk and network overhead for small file. Even for big file disk and network overhead will be there

[gurumoorthysivakolunthu]

Hi Sir... Shuffling Parameter -- is just the count, right... The count - number of partitions that can be shuffled between executors or Nodes between stages...
This is not the size of the partition, right Sir...?
Please help...
Also, please make a video about executors, drivers, tasks... i. e. Full life cycle Or flow of How spark job is executed ... Thank you, Sir...

[shwetankagrawal]

how spark.sql.shuffle.partition works here whcih is 200 by default because partition should be same as number of unique value in a key.