Synapse Espresso: Notebooks vs Apache Spark Jobs Definitions: which one should I use in Spark Pools?

Welcome to the 34th episode of our Synapse Espresso series! In this video, we will demonstrate and explore the differences between using notebooks and Apache Spark Job Definitions as authoring options in Synapse Analytics. We will cover the advantages and disadvantages of each approach and provide guidance on how to choose the right option for your use case.

Stijn Wynants - Fasttrack Engineer

Estera Kot - Program Manager

[avicool]

Pls share learning video how we can use Spark Job Definition for Stream Analytics

[moeeljawad]

Thanks for this video, i am currently using the notebook activity in Fabric Pipelines. My notebook is mature now and it runs very well. I was thinking of gathering the code in the notebook into a job definition, for the sake of saving execution time in the notebook. Would replacing a notebook by a job definition makes the code execution faster? Another question would be about job descriptions themselves, if you have defined some helper functions in the notebook, can i move them to a side job definition that is being called from the main job definition? if yes then how?
Thanks

[owenbolton]

how nicely you explained all the concepts!!good job

[aliceross]

watched the video thrice but did understand the concepts...thanks

[shawnglover]

the spark job definition is so deep to understand

[scottgipson]

thanks for introducing me to so many new topics

[agammishra]

but there is notebook activity as well, what is the issue if we schedule the notebook activity in the pipeline, it can run automatically as well, please comment on this

[patriciafuller]

I never knew about the web interface till this video...

[keenfive]

Are there any any noticeable disadvantages concerning execution performance of one over the other?
Does a scheduled Notebook take longer time until the actual execution of cells begins in comparison to Spark Job Definition? e.g. because Notebook need to initiate IPython interpreter?

[rosehudson]

that demo was needed to understand more precisely

[pakkalibrandu]

John and Jane are characters of Tarzan X, she picked up those name in demo 😆