20. Get Latest File from Folder and Process it in Azure Data Factory

In this Video, I discussed about how to get latest file from folder and process it in Azure Data Factory

Link for Azure Databricks Play list:

Link for Azure Functions Play list:

Link for Azure Basics Play list:

Link for Azure Data factory Play list:

Link for Azure Data Factory Real time Scenarios

Link for Azure LogicApps playlist

#Azure #ADF #AzureDataFactory

[vijaybodkhe]

I think Set Variable 2 (i.e PreviousModifiedDate) should have been inside If condition. Current file modified time should always compared with the highest modified time among previous files.

[MaheshReddyPeddaggari]

I am very much waiting for these kind of scenarios
Thanks Maheer

[sayonbhattacharjee]

Amazing Video...
Just wanted to cross check that I think set var (last mod date) should also come under IF activity. Then only it is working correctly in my case.

[SathishKotte-pz]

Hi Maheer, I'm learning ADF watching your videos, it's amazing series, I just want to cross check, I think we need to use 2 set variables in true condition only, need to put previous modified date first then latest file name variable, then only it's working fine in my case, Thanks

[mysahil]

Hi Sir, I am really liking your series of videos & learning from it. I believe there is one issue in above implementation as the setVariable2 should also come under if true condition only along with setting variable 1. It's working in your case since your latest file is the last file that runs in foreach loop but if It won't be last then it will not copy that file. pls check.

[Imrannaseem]

Thanks Maheeer I was looking for this video cheers!

[roshankumargupta]

Thanks for this video! Can you also create a video to explain how can we verify sources and target tables? Like how can we verify all the rows and columns value got copied correctly using data factory

[multipleaccounts]

@wafastudies thanks for your explanation. But this solution is not scalable right with the increase in number of files the for loop has to check all the files everyday to get the last file everyday. Any scalable solution would you suggest?

[prathaps]

Hi Maheer, Thaks for your efforts on doing this video's.its really very helpful.I am looking for similar scenario but instead of file need to get latest table records from SQL server..can you pls explain it how to get them..

Thanks

[abhishekrana]

hi sir thanks for such simple n very informative videos, Can you make one video on how we can resume failed copy activity from where it is failed not from the starting how we can achieve it ?

[aishwaryam]

Hello sir,
Can you please tell me what if we are getting two files with same last modified date and time?
What can be done for this

[mabunnicherukuri]

Thanks for this video, can you please share we have input is excel daily basis files, we want latest file name with last modified date, how to implement ADF pipeline

[varung]

Hi Maheer, Cant we have both the 'Set variables activity' inside the 'if condition' true activity?

[HnkBnnndk]

Nice and clear explenation. But when i try this on files in different subfolders, the mechanisme doesn't work with wildcards for subdirectories. Do you have a solution for that?

[dineshdeshpande]

Sir, can we sort the list that we got in JSON of the file son lastmodified date in DESC order and get latest modified file.

[MoHz-rxmy]

I think solution would be to use notebook PySpark or python? Am i right?

[gopalammanikantarao]

HI Sir, could you please help me out this requirement. how to get oldest file from folder and process it in azure data factory

[anmolganju]

What if I have a date level hirearchy in a data lake gen 2 where I have folder strutcure for each table as /table1/2022/01/03.. /table1/2022/01/10 and files are present there, now how should I pick latest file in this case?

[govardhanbola]

We are processing files from SFTP location. but the issue is each time we upload a new file in sftp location and run the pipeline, it's processing already processed files along with the new file. As the number of files keep growing, it is becoming a problem. instead what we want is, once the file processed, we want that to move to an archive folder in SFTP location so thatonly latest file will be processedin next run. how to do this

[joyyoung]

what file for the data set from very beginning? is it static csv contains all file name? cannot follow.