Part-3: Real time end to end Azure Data Engineering Project

Welcome to AnalytixCloud, In this hands-on tutorial, we'll dive into a real-time project on Azure Data Engineering. Whether you're an experienced data engineer looking to expand your skills or a newcomer to Azure, this project will provide you with valuable insights and practical experience.

This is the continuation to our Part-1,2. If you haven't watched it yet. We recommend to watch these videos before this :

In this video, we'll cover:

1. Data Ingestion:
===================================
We've designed a dynamic pipeline in Azure Data Factory that's versatile enough to handle both full load and incremental data loads seamlessly. By intelligently adapting to the specific data type and requirements of each task, this pipeline ensures efficient and scalable data integration, allowing us to maintain data consistency and minimize processing time for various data scenarios.

2. Control Table Mechanism:
===================================
A control table in an Azure Data Factory (ADF) pipeline serves as a dynamic configuration source, enabling you to centralize and manage parameter values, connection strings, and other settings for your data workflows, simplifying maintenance and enhancing scalability. By referencing this control table, ADF pipelines can adapt to changing requirements without needing code modifications, promoting flexibility and efficiency in your data integration processes.

3. Error log Capturing:
====================================
You can efficiently capture error logs through stored procedures in Azure SQL Database by creating a dedicated error log table to store relevant information. These stored procedures can be designed to log errors by inserting error details, timestamps, and additional context information into the error log table, making it easier to track and diagnose issues in your database applications. This approach streamlines error management and ensures that you have a comprehensive record of errors for troubleshooting and analysis.

You can find below the resources used in this video :

Mob: [phone]

Please Subscribe us for more videos:

Don't forget to like, subscribe, and hit the notification bell to stay updated with our latest Azure and data engineering tutorials. If you have any questions or need further clarification on any topic covered in this video, please feel free to leave a comment below, and we'll be happy to assist you.

Thank you for watching, and let's dive into the world of Azure Data Engineering together!

#azuredataengineer #dataengineering engineering #dataengineeringtutorial #endtoendproject

[AnkushVerma-tfhl]

Great explanation of how to build a dynamic pipeline in Azure Data Factory Looking forward to the next part on the bronze silver and gold layers

[alareqi]

can you please show us how you created the pipeline and the linked services, I am not able to follow :(

[muhammadqureshi]

I learned a great deal kindly share the remaining part of the video to help us to complete the project....thanks a lot

[harshadeep]

Waiting for Databricks other layers videos and about Triggers as well..
Great video btw :)

[healthy_wealthy]

I loved the videos . could you please provide more videos on data modelling ? will really appreciate that

[ashishgudla]

This is good but Instead of showing all developed functionalities id suggest developing in real time all the pipelines and linked services so that we can understand the exact flow.

[sathiyanarayanchakravarthy]

hey good content! Can you fix the audio an make it more clear and devoid of echoes and muffles in between?

[gqfkbgj]

how did you connect sql server with the pipeline....Is there any other reference for it??kindly let me knbow because iam getting error...i don't know if it has to do with my db...is there any other configuration to be done in SSMS???

[HemantKumar-suqt]

Hi sir
Hope you are doing well
I am an enthusiastic fresher data engineer. I want to create a data engineering project by taking a one month free subscription on Azure Cloud and show that project on my resume. If my one month free subscription on Azure Cloud expires and the resources get exhausted, will my data engineering project disappear or I will not be able to see it? Can I still show my data engineering project on my resume and the company can see it even after my one month free subscription on Azure Cloud expires?


Thank you so much

[smdimran]

Waiting for other layers videos till end and about Triggers

[ranjansrivastava]

Dear it's really very informative video. Could you pls share the code if possible. I am very much interested in error and logs.

[harshadeep]

Sir, may I know how to associate triggers with this pipeline?
I have created this generic pipeline as you shown in the video but my files or data in sqlserver arrives at different timings of day..
Let's say I have total 20 tables which needs to be copied to ADLS
For 5 tables data comes at 9AM
for next 5 data comes at 10AM
for next 5 data comes at 1PM
And for remaining 5 I don't know when data comes
So how to configure this
And as per this pipeline, since im keeping whenever i run it will trigger all my pipelines since tbey have indicator=1

[hamidabano]

Hi @analytixCloud - Thanks a lot for the end to end project. However I am unable to download the "datasets" from github. I tried downloading the raw file as well as tried copying them but none of it worked for me. The other files containign sql queries are opening up for me. But only the dataset file isnt opening/downloading for me. Could you please resolve this so that i can downlaod it and use it in my on-prem SQL DB?

[ashishgudla]

I did not understand the use of creating an azure SQL server resource...correct me if I am wrong since we are connecting to on-premise SQL server using a linked service from adls and then using a data factory to extract the data and store the data in Parquet format in the storage account..I don't understand why we need Azure SQL server

[kalki]

Hello Sir, Project is very helpful. But my suggestion as a learner, instead of creating pipeline early and explaining it will not be much efficient. Instead please create and show pipeline and all activities in video itself. It will be very efficient.

[ramswaroop]

How to get that datasets and import in on prem .... You didn't explained.. how can we do by ourselves

[Mehtre]

How many days will it take to upload all videos of this project sir

[prabhatgupta]

How will u update the watermark value to the latest one?

[vysaivicky]

Waiting for next part bro upload soon😑