8.2 Incremental data load in Azure Data Factory #AzureDataEngineering #AzureETL #ADF

[shanthababu]

Excellent approach, Please explain adequately to learners.

[trendy_techer]

Thank you so much sir for the valuable info

[teunjacobs]

Hi Cloud Guru!
First of all, thanks for the clear explanation. It really helped me in creating a incremental data load in ADF.
Never the less I was wondering if you could make a video of making the tables which should be synced incremental more dynamic. Now I've hard set in the lookup action that it should look for a specifc table. I've more tables which should be synced incremental, so I assume that should start working with an iterate activity like a for each(?). So that in a specific SQL table I've defined which table(s) should be synced incremental and that it does that automatically. Do you have any ideas of tips and trics for things like this? Thanks in advance!

[saipraneeth]

thank you for the video sir.. yesterday i got same question in the interview

[SantoshKumar-yrmd]

you should explain from very begining like when you created table in your datasource, nice

[chandandacchufan]

should explain how you are updating the table tbl_control

[harrydadson]

Please can someone please assist me on how I'm not able to do a Copy Activity to Stored Procedure on MySQL table in sink setting. Please help. Thanks

[alexcarter-jones]

Great video! Have you ever set up an incremental load between an Oracle source table and an SQL sink table before? I am currently trying to do this but can only get my delta load to work when copying from SQL to SQL as you did in this video, any guidance would be greatly appreciated :)

[souranwaris]

Hello Sir.
I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table and store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

[swaminoonsavath]

Expression of type: 'String' does not match the field: 'additionalColumns' I am getting this error .. MY source is Servicenow and Destination is AZURE SQL DB .. Please help me on this

[ssbeats]

Hi sir, in my source path i have a files i have created a shedule trigger to run every 1 hour. My issue is from source to sink the files are getting triggered duplucates files
Eg : last hour i have 10 files trigger nxt in my source path i recived 5more files. When it trigger the files of last 10+5 files are getting to the sink path

[camvinh]

why do we need control table for the last update time - since we can get it info in destination table

[muapatrick]

is it possible to add theses incrementals on parquet files on adls?

[vasistasairam]

The explanation is very good, but while trying to implement the same, getting errors, especially with formulas. Kindly make videos little more detailed.