12. How to copy latest file or last modified file of ADLS folder using ADF pipeline

In this video, we learnt how to get the last modified filename and copy that file to another ADLS location using ADF pipeline
#adf
#azuredatafactory
#datafactory

[ktdickinson]

Stack Overflow, ChatGPT, and Microsoft documentation couldn't easily explain this process to me. This YouTube video is gold - thank you for the detailed and clear steps! This worked perfectly and helped me understand the nuances of the ADF pipeline.

[anoop]

Thanks for a great step-by-step explanation of how to copy files by the last modified date.

[ZacClark]

Awesome video and very helpful, thank you!

[aadilmohammad]

I like the way of teaching and content

[mohank.]

It is Really helpful annu. I suppose to start like this. But dont have proper time. Anyways you done a great job. My self Anil. I’m a solution architect in data engineering

[johnpaulprathipati]

Very good explanation 🙂 many thanks ANU.
i am following your each every video keep going....

[dataisfun]

This is Golden thanks a lot you just made my day.

[pyspark]

very helpfull, but walk through bit slowly

[alokkumar-dbvw]

HI, Thanks for the detailed steps .Its a very good content and its working also. Could you please share how to load latest file to snowflake table from blob storage ? Please note the file is excel one.

[radhikakamalesh]

Clean and very simple explanation. Can you please let me know how I can send the latest file as a attachment in a email instead of copying it to another folder

[ippilivenkataramana]

Good explanation.
Small advice for you to make it more useful this video to viewers is write the steps while implementing or before starting the implementation. Then learners can easily remember the steps and process. Don't think this is -ve comment. Just explored my experience after watching your video. I have seen lot of videos in other YouTube channels, no one is explaining the implementation with writing steps. Beginners like get confusing with the steps. If possible to you then you can add the steps. This is my suggestion only

[jovianaditya]

Hi Annu, I have two parquet files in my storage. After I run my pipeline why am I getting two set variable2 and two set variable1 as the pipeline result? It is supposed only to show one set variable1 and one set variable2 right as the result for the latest modified file?

[varshiniparamasivam]

Great video, I am having this doubt here in this video we are getting latest file in dataset from there can we use that lastest file in dataflow

[ashishgochhi]

Is it possible to apply same in SharePoint to extract the new/modified files..?

[hosseinj]

Thanks very much Annu. it's been a very helpful and handy video. It helped me to get the latest file in a blob storage. I have a question however, if I have a mix of different file types (lets say csv and JSON), is it possible to specify to extract the name of only the CSV file? I created a blob>eliminated csv dataset, however it seems to select any file times in the directory. I would appreciate it if you could advise how to sort this issue out. Thanks

[akshataprabhu]

hi annu...i am looking for the same opertaion to be performed using azcopy command in a shell script

[husnabanu]

Hello your videos are awesome..in real time scenario what if we are moving the files after copying to archieve folder this way source folder will have only latest files..should we still consider using this approach??if we have more number of files it will take huge time and performance may impact?please advice

[bolisettisaisatwik]

Would the set variable capture two files with same last modified data and two different file names?

[gowrishankart]

Hi Annu, if i want to compare current date and lastModified date then what would be the if condition expression?, If yes thn success els failure..

[aaravkumarsingh]

But this only work for single file, like when i upload more than one file Ex.- 2 file, then it only fetch latest uploaded file not both files, how can i handle this ?