3. Incrementally copy new and changed files based on Last Modified Date in Azure Data Factory

In this video, I discussed about Incrementally copy new and changed files based on Last modified date in Azure data factory

Link for Azure Functions Play list:

Link for Azure Basics Play list:

Link for Azure Data factory Play list:

#Azure #ADF #AzureDataFactory

[chinmaykshah]

Good video can prepare real time scenario where incremental read from one json file and post data into two different azure sql tables which having relationships between them as well.

[prashantshetty]

Hi,
Thankyou for the amazing videos it has been a great help.
if you can make a video on below scenario it will be really helpful.
if there is a video on this kindly help me with the link
Scenario : 1. Copy files from blob to Gen2.
2. Making sure there is a retry mechanism if the pipeline fails and then copies only the files that were not copied before .

[kelvink]

Thank you. Your training video is helping us to learn quickly

[HariKrishna-cjuq]

we don't know the date of the last update files. how to do that file in an incremental process?

[varunkulkarni]

Very Useful Video, Appreciate your work

[vishalraj]

simple and to the point explanation. Great!!

[lukabirtasevic]

Thank you so much. Very helpful and clear explanation!

[pawanreddie]

Sir, please do a video on incremental load from sql to storage..please!!

[shaileshsondawale]

Wonderful efforts.. !!! You made our life easy.. :)

[ritujavyavahare]

Thank you for the video. Is this scenario can also be completed using the Getmetadata activity too which you have explained in another video using field list ->Last modified

[vijaysekhar]

Sir How should we load incrementally if, instead of days HOURS/MINUTES are given?

[balajia]

Thanks bro! very nice. Have you done the incremental load for ADLS Gen2 when there are multiple folders?

[annekrishnavinod]

for example if we receiving files in every one hour, how to load latest file? is there any option/setting to ascending /descending the dates?

[ArjunyadavArya]

HI, I faced an issue and let this sort out in a beautiful way;

Q) If the EMP table is in server 1 and the department table is on server 2, how do we copy that into data lake storage in Azure Data Factory by using only one activity andonly one pipeline?

[niharikasmily]

Hi sir, excellent explanation. Can you please tell me if in a folder today morning one file is uploaded, today afternoon another file is uploaded. How to copy the latest file i.e only afternoon file not morning file?

[happybake]

How can we do based last 30 min updated files

[masoudghodrati]

Thanks for the vid! Very helpful.
I have a question: how is the performance of incremental copy? Imagine a scenario that a directory gets millions of files every day. Does this mean that the pipeline first checks the last modified date of all files every day and then filter those new ones and then copy?
If that's the case the performance can decrease as time passes.

[repalasanthosh]

any video on the same topic but the copy is between azure blob storage and synapse tables? please reply

[lakshminarayana]

Can you do video on SQL database, how to load incremental load

[ObjectDesigner]

is it possible in ADF to run a mapping data flow when (trigger) a file is created in a folder or is it something which can be done in Logic apps only, by from Logic App trigger ADF ?