Incrementally copy new and changed files based on LastModifiedDate

In this tutorial, you'll use the Azure portal to create a data factory. Then, you'll use the Copy Data tool to create a pipeline that incrementally copies new and changed files only, based on their LastModifiedDate from Azure Blob storage to Azure Blob storage.

By doing so, ADF will scan all the files from the source store, apply the file filter by their LastModifiedDate, and copy the new and updated file only since last time to the destination store. Please note that if you let ADF scan huge amounts of files but only copy a few files to destination, you would still expect the long duration due to file scanning is time consuming as well.

Note

If you're new to Azure Data Factory, see Introduction to Azure Data Factory.

In this tutorial, you will perform the following tasks:

Create a data factory.
Use the Copy Data tool to create a pipeline.
Monitor the pipeline and activity runs.