Databricks Real Time Project- Preparing for Production: Automating Databricks Notebook-Part 1

Hello Everyone, I am Naval Yemul. Welcome back to Data Master!
Follow me on LinkedIn:
In this video, we dive into the essential skill of productionizing and automating Databricks notebooks, perfect for anyone looking to upskill in Databricks or add impactful projects to their resume. We'll walk through a real-world example from one of our e-commerce clients on Amazon, who provides raw CSV files in Azure Data Lake Storage (ADLS).

Here's what you'll learn:
Ingesting raw data into the bronze layer.
Automating the ingestion process using Databricks workflow jobs.

If you enjoy this video, please share it with your friends and colleagues, like the video, and subscribe to the channel. Have specific topics or questions? Leave a comment below!

Check out the medallion architecture and other related videos linked in the "i" button.
Links:
Medallion Architecture Explained:
Managed and External Tables:
Databricks Playlist:
Databricks Certification Playlist:

0:00 Introduction to Real Time Project/ Understanding clients requirement.
3:22 Databricks Workspace
4:16 Understanding Sample Amazon Dataset
7:10 Uploading raw data in ADLS container
8:37 Exploring External Location/ Storage Credentials
10:18 Preparing for Production Notebook
15:11 Renaming column name using toDF
17:17 Writing to External Delta table
23:31 Explaining about Widget
24:31 Creating widgets
29:03 Removing raw file
29:28 Uploading new file
33:24 Schedule the notebook

[abhijeetab]

Hi Naval, To me, you are a Databricks Hero. I have learned a lot related to the Databricks platform and successfully cleared multiple Databricks certifications. My sincere request is that you provide a full end-to-end project explanation. It would be great for all of us to learn and grow by watching your amazing YouTube channel.

[ShahnawazAnsari-ud]

Thanks for the video really helpful,
plz make a video on how the complete Orchestration happens in Azure data engineering,
how adf pipelines and db notebooks deployed to prod.

[abdulmalik-emlb]

Hi First of all Thank you very much. You are doing great work I learn lots of things through your videos...plz make the videos #real time projects for #Databricks, #ADF with connectivity and access part also if possible. 😊

[JeevanSandesh-wl]

Great tutorial on automating data ingestion with Databricks notebooks! Learned a lot about streamlining data workflows

[karthikrajanatarajan]

Nice Explanation looking forward for more projects videos. Tq

[harshshah]

Kindly make video for end to end data engineer project.. I learn so much from your channel and recently cleared first round in one of fintech company.. thanks for your hard work so we can see quality videos… awaiting new end to end .. once again thank you

[muhammedshafik]

This is great to learn with real data. Thanks for the video. Expecting more videos on this topic.
Have one doubt: How a developer/tester can validate the correctness of cleansed data loaded from CSV to Raw layer if the file is removing immediately after loaded and processed to next layers?

[mmhuque]

Great Video, I want to see more from you

[AmarKumar-kwqy]

total project explanation will be great

[janardhanreddy]

thanks you very much, Wonderful Job Brother, Please upload the remaining parts

[ShivamSingh-foue]

Hello Naval
Shivam This side.!
Thanks for explanations

[gudiatoka]

Please make a project using azure databricks

[vasiminamdar]

hello naval make some videos on real time adf and adb scenarios asked in interiew

[Halikal_Veera]

Please Azure Engineering, Data Factory and DevOps and Azure databricks and finally load into gold and reporting how it has done..Please create video for this to under stand better in cloud projects

[ritwickdey]

Can you please make end to end total project video

[jayanthreddyvallem]

Please upload end to end process of data engineering where we dont have to go anywhere we just need only the full process uploaded from scratch please