Incremental Data Loading from an ERP System into Azure Data Lake Using Azure Data Factory

Created By: Muditha Pelpola

This is my 5th YouTube video and this time I decided to talk about Data Engineering related topic .

In this video session I talked about how to push incremental data into Azure Data Lake using Azure Data Factory.
Here I selected an ERP system (Infor m3) that I worked with more than 5 years. I got business problem to push M3 ERP tables into data lake
with a file structure “ M3 File Name Year  Month  Date ( Based on M3 Last modified record) “. Without hardcoding values in Data Factory
to create file structure automatically based on M3 Last modified date was the challenge .
I solved it using Data Factory expression and this video is to share my experience with data community.

[vemarajulasya]

really good content. thanks for the video. and please upload more videos on data loading from a particular source to data ware house step by step.
please!!

[anujgupta]

please upload more videos on azure data factory incremental loading using merge. and how to use dynamic expression s

[dileepkanumuri]

Nice video! I have a doubt. Will ADF handle incremental changes on the source DB like Deletes and Updates to the records?

[khana]

what if I have multiple table under my Copy activity using for each loop, and each table have different column to identify delta, is that possible or this has to be one table copy at a time

[anthonberg]

Great video! The code for the demo M3 database is not something you would like to share? Thanks!

[anujgupta]

if daily data is coming in my blob storgae I need to insert in my sql tables how to do it automativally. I have 10csv and each csv contains each table details i.e 10 csv and 10 tables in my azuresqldb need to be inserted.

[gauravmodi]

If I want all the incremental data in a single file like lets I have done a full load of a table "users" now I want that the Incremental data but in the same file not indifferent date wise file so is that possible.