An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer

#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault

This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.

The tools that are covered in this project are,

1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI

The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.

Follow the below link to import the database which is used in this project to the SSMS (I used the light weight version)

Chapters

TimeStamp:

0:00:00 - Project Overview
0:11:02 - Environment Setup
0:19:40 - Data Ingestion using ADF (Part 1)
0:41:31 - Data Ingestion using ADF (Part 2)
1:01:42 - Data Transformation using Databricks (Part 1)
1:16:22 - Data Transformation using Databricks (Part 2)
1:36:26 - Data Transformation using Databricks (Part 3)
1:51:58 - Data Loading using Azure Synapse Analytics
2:15:34 - Data Reporting using Microsoft Power BI
2:30:35 - Creating Security Group using Azure Active Directory (AAD)
2:36:44 - End to End Pipeline Testing

– – – Book a Private One on One Meeting with me (1 Hour) – – –

– – – Express your encouragement by brewing up a cup of support for me – – –

– – – Other useful playlist: – – –

– – – Let’s Connect: – – –

Instagram: mrk_talkstech

– – – Tools & Equipment (Gears I use): – – –

Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!

– – – About me: – – –

Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"

I will be creating playlist which covers the below topics (with DEMO)

1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)

After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,

1. Data Engineer
2. Data Analyst
3. Data Scientist

Can't wait to help people with my videos.

– – – Support me: – – –

[mr.ktalkstech]

I hope you find this video useful. If you have learned something or found it helpful, please consider giving it a like.

What topic should I cover in the next end-to-end project video? Please let me know your thoughts in the comments.

Follow my LinkedIn profile to get regular updates:

Thank you!😊😊😊

[harrybassi]

This is the best end-to-end project I have ever seen. I was able to set up a whole data engineering solution and personally implement it in my company based on this video. Please add more end-to-end projects when you get time. Amazing tutor and amazing channel.

[kalicharankhetwal]

This is the video I have been searching for a long time to kick off my databricks journey, now Mr. K you offered it for public free, thank you very much. :) Please keep it up.

[bruhlickd]

The BEST tutorial about data engineering that I have Found. I'm a data analyst, but in my company I'm having to do a LOT of data engineering job and I was totally lost till I found this tutorial. THANK you for helping me.

[shwetabhat]

This channel is definitely my go to place for End to End Projects . Absolutely love your content sir . Keep it coming . Thank you 😊

[drivetrainerYT]

Crispy clear, no-nonsense, dense, structured. Definitely subscribed.

[pramodmaurya]

First time, I've encountered a comprehensive and practical explanation from start to finish. Thank you very much for all your efforts.

[ehenrysixteen]

A senior colleague recommended this video to me and I must say I am beyond grateful. Thanks for compiling such an amazing content. A Game Changer!!!

[vijayalakshmiv]

Great video! Your explanation of End-to-end Azure Data Engineering was clear, simple and easy to follow. I appreciate your expertise in breaking it down for us. God bless!!

[somasekharreddynakkala]

Mr. K, you are the gem man. Your way of explaining the things superb. Thanks a lot for this free content.

[caesardutta]

Mr. k Sirji I went through the first part of your video - Environment setup. You mentioned that after going through end to end one can have the architecture in one's resume.

Surprised that basics like Linked Service, Datasets were just created on the fly. These are extremely important components of ADF and no concept was provided. I am lucky that I have previously gone through certain ADF data movement tutorials

All essential components were already created and creation is services forms a basis of any tutorial.

You create a SHR IR service for one premise SQL Server but the most important thing is the configuration of Security Lists / Groups (Virtual Cloud Nw Firewall).

Anyway the presentation is good, voice volume is optimum and speed is also okay.


Namaste 🙏

[bjoe]

Please bring some more End-to-End projects on Azure

[overlord]

Watched the tutorial in one go. When i decided to become azure data engineer and tried to watch your video it looked me difficult bcz azure have so many services with all those authentication. Then i practiced the adf, data lake, databricks, powerbi for months taking DP-203, PL-300 exams and making some small projects then i understood this project. For a beginner this might be difficult.

[sharathchaluvaraju]

Hats off to your patience and the detailed explanation of end to end pipeline, even the paid content courses will not give you this level of explanation ..grateful and thankful for this amazing end to end project video

[ashokvenkat]

Your presentation and explanation were excellent, and Microsoft owes you appreciation for it!!

[devinpikafire]

What a powerful 1st 10 minutes... Looking forward to following along! I am just a SQL DB to Power Bi, but want to push the company forward and this is just the knowledge I need to begin :)

[riteshtripathi]

the videos thati have gone through both from Udemy and Youtube, let me share that this content is far better than any of the videos i have gone through. He explains in deep how and why. Loving it

[PawanSharma-ddjv]

Excellent and well described end to end data pipeline. Waiting for more videos on real time and incremental batch processing using ADF and databricks. Thanks

[abhishekanand]

This is the most action packed azure project i have seen till date. The instructor is really very knowledgeable. This project itself is far more informative than most of the paid course on ADF.

[mayurjadhav]

Thank you MrK, I’m a project manager with not so tech background and I look after these kinds of projects. Your explanation gave me perfect idea of understanding the flow functionally and made me aware of Data engineering projects end to end, thank you!