Databricks Workflows Jobs - Automating Databricks Notebook- Part 2

Hello Everyone, I am Naval Yemul. Welcome back to Data Master!
Follow me on LinkedIn:

In this video, we delve into Databricks workflows and jobs. This video is a continuation of our Databricks project playlist/podcast. Part 1 covered preparing for production with Databricks notebooks. In this video, we'll explore Databricks workflow jobs, including task configurations, trigger types, and scheduling. Finally, we'll demonstrate how to schedule these jobs.

We'll discuss real-time use cases and scenarios. If you have any specific doubts or questions, please leave a comment on this video.

Links:

Preparing for Production: Automating Databricks Notebook - Part 1
Complete Databricks Playlist:
Databricks Certification Playlist:

0:00 Introduction to previous video on Preparing for Production ready notebook.
1:00 Overview on Databricks Workflows
2:51 Databricks Workflows Jobs
3:29 Types of Databricks Workflows: Jobs and DLT
4:51 Create Job and talking about Scenario
7:00 Task configurations
12:42 Explaining Notebook and widgets
17:26 Databricks Runs
19:08 Managing Dependencies (Theory)
20:44 Job Details/ Job Configuration
21:52 Schedules and Triggers
27:15 Triggered
29:47 Output
31:20 Thanking you

[sufiyaanbhura]

Thank you so much for the demo! Eagerly waiting for the next parts of this project.

[lalithguptha]

The Databricks playlist is really helpful for preparing for the certification. Ur resources helped me solve the 5 new questions we get and passed with 100 percent

[JeevanSandesh-wl]

This tutorial is so helpful.Thanks for explaining it so clearly!!

[sumit_prakash]

Please make a separate video on Delta Live Table.

[mmhuque]

Thank you for the detailed explanation in part 2. Could you please demonstrate the process with a file arrival? Do I need to set a parameter name for the file upload? For new file uploads, can we automate this process without needing to add the file name as a parameter input?

[ShahnawazAnsari-ud]

Very useful, Thanks for the video.
could you please make part-3, show all layers e.g silver, gold.

[chinchud]

Can we make the parameter dynamic instead of giving the value manually?

[kamlakarkamble]

can v run multiple notebook in parallel on same cluster ? if yes how (shall we do just by keeping dependency tab blank ) ? pls answer