Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark

Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark
In a data integration solution, incrementally (or delta) loading data after an initial full data load is a widely used scenario. The tutorials in this section show you different ways of loading data incrementally by using Azure Data Factory.

In this case, you define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark.

00:00 Introduction -Agenda of Demo
03:30 Create Azure SQL Database
05:30 Create Azure Data Lake
06:15 Create an Azure Data Factory
07:35 Create required tables/SP in Azure SQL
10:00 Create Linked Services
12:40 Create ADF Pipeline
20:30 Run ADF Pipeline
21:30 Verify Azure Data Lake for File
22:40 Update Azure SQL Records
24:20 Run ADF Pipeline
25:10 Verify Azure Data Lake for Delta File

Please follow and ask any question to our linkedin profile and twitter or our web site and we will try to help you with answer.
Linkedin
twitter
website
FB

Here Group of People are sharing their Knowledge about Software Development. They are from different Top MNC. We are doing this for community. It will help student and experience IT Pro to prepare and know about Google, Facebook, Amazon, Microsoft, Apple, Netflix etc and how these company works and what their engineer do.
They will share knowledge about Azure, AWS , Cloud, Python, Java,.Net and other important aspect of Software Development.

[palyamsudheer]

Great content, Thanks. How do you deal with deletion of records?
I have an OLTP server (SqlServer) where application can delete records, so trying to understand 1. how to identify delete 2. Treatment for deleted records in the late (something like making them inactive or highenddated etc)

[SandhyaVeda]

Can u please mention the syllabus, which is sufficient to get job in ADF. Also kindly order ur playlist

[souranwaris]

Hello,
I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

[ranjansrivastava]

Good Article, Lack of Visibility of the Video.

[manishkumar-hoiz]

what the fuck....how can lower lastmodifytime can pass through it..meaning the copy condition gets false and data should not be copied