Azure Data Factory - Incremental load or Delta load using a watermark Table

Azure Data Factory - Incremental load or Delta load using a watermark value

Here source is having incoming values with some date value, we are keeping a watermark table which keeps track of the data copied and this helps to incrementally load the data to the destination preventing copy of the old data.

Refer below to setup self hosted IR to use SQL from local Machine:

To copy multiple SQL Tables incrementally, please watch:

Check out my Udemy Course on Building an end to end project on Azure Data Factory and Azure Synapse Analytics

Udemy course with Coupon link:

My New course on Complete Azure Synapse Analytics:

Limited time to enrol this course with above Link for can access the course with 50% OFF!!!

============================

============================

[DidaKusAlex]

thanks for your explanation, it was easy to understand.

[shanukatiyar]

This is really very helpful video. Thanks bro

[KaraokeVN]

Thanks for your videos, keep going bro

[prabhakarredekar]

thank you for this video can you share that text file link as an attachment for our information

[camvinh]

Hi, This video very cool. I just have a question, why we need a watermark table for the last update time of the target table, because in the next pipeline run i think we could query target table and get max_date from it. Thanks for your support

[MarianaRodríguez-en]

How can I do the last step if I want the table to be connected to another database instead of ADLS? It's my first time doing something by my own in DF, help.

[kumarssss]

Thank u much.. wat to do it there was update? Rather than insert

[CameramanHQ]

Nice video, I wonder if this could also work if you update an existing record instead of adding?

[viniciusandrade]

Hi, in my case I would like to use this approach for historical load portioning folder in Blob storage by Year/month using watermark column/value instead of utc()now . Is it possible? Do you know any similar approach?

[souranwaris]

Hello Shanmukh Sattiraju,
I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

[rajageetha]

Sai appa enaku nalapadiya data engineer job kedaikanum... Intha muyarchi nega kuduthatu sai appa ipo na nalapadiya antha muyarchi iruthiku vnthuten nega ena kutitu vnthutinga appa. Interview atten pani nalapadiya velaiku poga nega ta arul puriyanum appa. Ungala matu ta namburen appa . Husband ku NLC nalapadiya velai kedaikanum. Naga tamilnadukae thirumbi varanum sai. Elame en sai appa aasi odum arulodum nalapdiya nadakum nu namburen appa. Elame en sai appa . Nalathe nadakum sai appa. Om sai ram jai sai ram 🙏🙏🙏🙏🙏❤❤❤

[mohdsaudahmed]

Suppose if i am unable to load the data from sql server from last 4 days and today i want to load that 4 days data, so how i can load it incrementally using the same adf pipeline.

[rohanpillay]

There is no audio we can hear, can re-upload this video again 🙂

[prabhatsingh]

Video has no voice, can you please look.

[shafeeqniyas]

nice explanation could you please send us the text document

[naveenyarra]

Hi, how can we implement incremental pipeline if we Have multiple tables in our source, please let me know

[rohitkulkarni]

I have 1 requirement I am passing the parameter in notebook1 :
%run

and calling this in Notebook2 and wanted to call $VarA .The "source_full_object_name" iis a delta table i need to run as selec * from source_full_object_name

Is there anything video related to pass the parameter in another notebook.Please share me

Thank in advance

[sanjayparakh]

what if my source table doesn't have date column

[ashwinreddyboppidi]

Thank you for the great video, Here I have one question, what if we have source and destination columns are different when we choose multiple table incremental copy ?

table1_Source table1_destination

Id ProductId
Name ProductName

Table2_Source Table2_Destination

PId ProId
PName ProName