Part 1- End to End Azure Data Engineering Project | Project Overview

#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault

This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.

The tools that are covered in this project are,

1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI

The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.

Part 2 will be uploaded soon. Stay tuned.

– – – Book a Private One on One Meeting with me (1 Hour) – – –

– – – Express your encouragement by brewing up a cup of support for me – – –

– – – Other useful playlist: – – –

– – – Let’s Connect: – – –

Instagram: mrk_talkstech

– – – Tools & Equipment (Gears I use): – – –

Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!

– – – About me: – – –

Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"

I will be creating playlist which covers the below topics (with DEMO)

1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)

After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,

1. Data Engineer
2. Data Analyst
3. Data Scientist

Can't wait to help people with my videos.

– – – Support me: – – –

[seedhiBaatNoBakwas.]

Great playlist for someone who has zero knowledge on ETL/AZURE. Good to clear fundamentals of azure resources

[madhavtn]

I just started doing projects on data engineering and to be honest, this series needs to be on top results. Very useful content and easily understandable to newbees. Eagerly waiting for new projects using new tools and cloud services

[RukshanEdirisinghe-vq]

this helped me to find my job. Thank you

[ZombieHelion]

Hi Mr.K your lessons give the view of the roles of Data Engineering. I really appreciate your videos and would like to thank you sir. May God bless you and your family.

[jabalraji]

Am a starter in DE....your illustration is awesome and I have subscribed to page for more updates...

[charankatta]

hi, great tutorial and indeed good learning for starters as me. Can you also please make end to end azure data engineering real time project with continuous data stream & readily available big data (so that we can readily download from your link). It would be of great help for us.

[Dataenginner]

The concept you just showed in 11 mins is more worth then others playlist 😂, good to be ur subscriber man ❤ please keep making videos and help student

[prabhatgupta]

wow what a explanantion ..Huge respect.
keep doing u will have good followers soon

[helovesdata]

we are using the Medallion architecture at my job now.

[prabhatgupta]

Plz plzz bring more.. U teach very well

[vamshiikrishna]

Please carry with more viedos your knowledge sharing is helping us a lot🙏

[Win_whatsimportantnow]

This video series is a game changer for me

[mansouralshamri]

Why do we use Databricks? Azure Synapse Analytics does ETL.

[hashmatsulthana]

Thank you so much for this content .can you also please bring up video for ADF to snowflake?

[atulbisht]

Sir this usecase doesn't make sense. They would want to eliminate/cut the on prem data warehouse to azure environment then why wiill we be connecting to it. For one bulk loat it is understandable but for daily refreshes the source should be an OLTP system?

Still thanks for making this playlist...it is really helpful to understand important azure services.

[Dipsvloggermany]

Can you make an end to end project using Microsoft Fabric ? And please make more end to end to project like this

[justvenkyy...]

such a good explanation. great work. please post on complex challenges that faced by data engineers and its solutions.

[tao-adl]

Awesome, some key concepts finally clicked in my brain. Great breakdown!

[dhruba]

Amazing content, Thank you for sharing this video series..

[chubsmash]

Thank you for these videos, really appreciate the time and efforts.