89. Databricks | Pyspark | Notebook Scheduling through Event Based Trigger using Azure Data Factory

Azure Databricks Learning: Notebook Scheduling through Event Based Trigger using ADF
================================================================================

How to schedule Databricks Notebook through event based trigger using Azure Data Factory ?

ADF plays pivotal role in scheduling various ETL pipelines through ADF activities. Azure databricks notebook can be scheduled through notebook activity in Azure data factory. Event based trigger creates job scheduling as soon as file arrives.

To get through understanding of this concept, please watch this video

#ADFEventBasedTrigger, #DatabricksNotebookScheduleADF,#ScheduleBasedTrigger, #DatabricksJobScheduling ,#DatabricksAzureDataFactory, #DatabricksADF, #DatabricksASQL, #DatabricksADLS, #DatabricksKeyVault #DatabricksOutputParameter, #DatabricksDbutils, #DatabricksNotebookExit,#DatabricksADFRunOutput, #NotebookActivity, #ADFNotebookActivity #NotebookActivityBaseParameter #DatabricksNotebookActivity, #ADFtoDatabricks, #NotebookViaADF, #DatabricksScheduling, #AzureDataFactory, #DatabricksIntegrationwithADF, #ADF,#PysparkTips, #DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[aniketjadhav]

Hi Raja, your content is very good and we can easily understand the concepts. Please continue the good work and make more videos on Databricks and azure adf

[AdityaTyagi-cm]

While creating trigger 9:55
In blob path begins with you passed event/trigger
But folder name is event_trigger
How did it got invoked ?

[rawaaelghali]

what if i have sub directories like i have a container and under it i have to check if a file is created in one of the folders

[shubhamsingh-jq]

Hi, the trigger is not invoking the databricks notebook. What can be the reason.

[rajunaik]

Hi Raja, how this approach is different from databricks Autoloader. I agree Autoloader is more for streaming data. Any other difference?

[alluchandrasekhar]

Can we do the same task without using databricks?

[rajasekharamuru]

Hello sir will you provide training on Databricks course
Hope you could reply please.

[nehalalex]

if more than one file is loaded at one point won't code break because you are giving lastfile[0]