49. Databricks & Spark: Interview Question(Scenario Based) - How many spark jobs get created?

Azure Databricks Learning:
==================

Scenario Based Interview Question: How many spark jobs get created while reading CSV file with different options?

This video covers more details about spark csv reading scenario. This interview question is based on real time scenario.

#DatabricksScenarioBasedInterviewQuestion, #SparkScenarioBasedInterviewQuestion, #DatabricksReadCsvInterviewQuestion, #SparkJobs, #NumberofSparkJobs, #DatabricksSparkJobs,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion #BigDataInterview #PysparkPerformanceTuning #PysparkPerformanceOptimization #PysparkPerformance #PysparkOptimization #PysparkTuning #DatabricksTutorial, #AzureDatabricks #Databricks #Pyspark #Spark #AzureDatabricks #AzureADF #Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial #azuredatabricks #notebook #Databricksforbeginners,

[nic-ivds]

Excellent video, would request you to do separate series on debugging the performance with UI, and how to debug physical plan, and UI options

[TusharKakaiya]

Excellent Question and up to the point satisfactory answer.... Kudos !

[navnathjarare]

very useful please create one playlist for Scenario Based question answer thanks

[nic-ivds]

df_no_option took 7.76 sec (1 job)
df_infer_schema took 3.04 sec (2 jobs) why is that ?

[agent_Vergito]

I think for both 1 & 3, 0 jobs should be created becoz jobs means no. of actions called. So in both no actions are called so 0 jobs should be created. Pls correct me if I am wrong

[suvratrai]

Hi Sir. Even in the 3rd option, it would still read the file at least once, isn't it? So should 1 job not be created here also?