Azure Data Factory - Copying Today's files with date and timestamp in name of the file

Azure Data Factory - Copying Today's files with date and timestamp in name of the file

If the incoming file contains the file format as date with timestamp and if we have any requirement to copy the data based on the Filename we can follow this approach.

If we want to copy the data based on last modified file date,
please follow the below approach:

If you want only non-empty file and modified or created date with today,
please follow below

Check out my Udemy Course on Building an end to end project on Azure Data Factory and Azure Synapse Analytics

Udemy course with Coupon link :

My New course on Complete Azure Synapse Analytics:

Limited time to enrol this course with above Link for can access the course with 50% OFF!!!

[amritatripathi]

Thank you @Sanmukh Sattiraju sir. Great help.

[aswinsimhadri]

Thank you so much,
I have a question i have file name with date extension, want to add an extra column stamping the file date, can you please explain how ca we do that

[rkvkg]

How to copy previous weekday file (ie) if I run on Monday it should pick Friday date file

[azzu]

Hi,

I have one query- which dataset you have used in copy activity for source

[apurvjadhav]

please can you explain (substring (item).name, 0, 10) in this what is 0, 10 does

[dataisfun]

Hello,

Thanks for this use case, but in a scenario where I want to consider the date modified in Azure Data Factory but I want to look at only the 'yyyy-MM-dd' ignoring the time who will I go about it. Because I have multiple files uploaded the same day but not necessary the same time.

[akshaykaushal]

What if we want to fetch full name along with current date only ?

[hanimireddyk]

Hi, I am testing this scenario, but seeing some challenges while doing with substring and utcnow() equal check.

Ex: file name used: emp_2022-12-05_123.csv
At condition check given as below: taking only date part for comparision, but this is not doing copy operations.
@equals(substring(item().name, 5, 10), formatDateTime(utcnow(), 'yyyy-mm-dd'))

Later with the help of set variable: @concat('emp', '_', formatDateTime(utcnow(), 'yyyy-MM-dd')) and then pass this as below

at condition check: @equals(substring(item().name, 0, 14), variables('date'))(In this way it is working fine).

Copy not working in below individual checks :
@equals(substring(item().name, 5, 10), formatDateTime(utcnow(), 'yyyy-mm-dd'))
@equals('2022-12-05', formatDateTime(utcnow(), 'yyyy-mm-dd'))

Not able to figure it out the problem here. could you try out the reason, much appreciated for your work.