72. Databricks | Pyspark | Interview Question: Explain Plan

Azure Databricks Learning: Explain Plan
========================================

Explain plan describes the internal details about query execution. When a query is submitted for execution, it goes through many internal steps to build efficient execution plan

This video talks more about those internals. This is one of the common interview question and also plays importal role in troubleshooting and optimizing code.

#ExplainPlan #ExecutionPlan, #DatabricksExplainPlan, #PhysicalPlan, #AnalysedPlan, #ParsedPlan, #OptimizedLogicalPlan, #UnresolvedLogicalPlan,#PysparkExplainPlan ,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[omprakashreddy]

I requested this a few days ago and here you are with awesome explanation. This shows how much you care about the community :)

[premsaikarampudi]

Honestly, I was intimidated my the length of the video but then after watching it. It was very simple :) Thanks

[Aramakishore]

very nice explanation raja..Thank you for your efforts to make this video..looking forward to more videos ...It is very useful

[amazhobner]

Hands down best df.explain() video I've watched so far.

[sonuamrith]

Wooow awesome super marvelous or if there are any other words in English which can praise in depth of concept you have explained.. Thank you very very much sir

[at-cvky]

Why is exchange happening after sort-merge join ? I see both the tables are initially exchanged and then sorted which means same DEPT-ID records are on the same partition that enabled sort-merge join. So, I can't understand why exchange partitioning is done after sort-merge join.

[pankajchikhalwale]

I have a question - please tell me whether my question is right or wrong.
.
I have a pipeline in databricks and it runs 4 times in every 24 hours in production environment. 8am, 2pm, 8pm, and 2am.
At those 4 times - there may be other different applications also running in the same databricks production cluster.
Say - at 8am 10 other jobs are running, at 2pm no other job is running, at 8pm 5 other jobs are running, and at 2am 4 other jobs are running.
If after all the logical and physical planning is done and if multiple physical plans are created and cost model is applied - then - will there be different physical plans, which will be executed at 8am, 2pm, 8pm, and 2am respectively ?


In other words - will the choice of physical plan actually executed at 8am, 2pm, 8pm, and 2am depend on actual run-time work-load/volume of work at that time (i.e. 8am, 2pm, 8pm, and 2am) in production environment ?

[tinashechinyati]

This is great content. Is it possible to version control data frame query execution plans?

[shilashm]

Vera level thala!!(G*d level explanation)