Microsoft Fabric: How to append only Incremental data using Data Pipeline in Lakehouse

This video will discover the ways to run Incremental ETL(Data Load) to append new data using Data Pipeline without using SQL Procedure. Lakehouse does not support reading and writing using SQL, then how to append only new data. In this Video I have not used Data Flow Gen 2

00:00 Overview of Incremental ETL
02:20 Create Data Pipeline with Append option
08:20 Create Data Pipeline for Incremental Data Load
16:30 Test the Data Pipeline

Last_date =
select max([collectdate]) maxDate from [dbo].[incr_info]

max_date =

🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.

With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.

Don't forget to like, comment, and subscribe for more great content!
▶️Data:

---------
▶️Follow me on:
---------
▶️My Other Videos:

-~-~~-~~~-~~-~-
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
-~-~~-~~~-~~-~-

[brianmorante]

I have been looking for this. You explained this so I can easily understand! This helped my team. Thank you.

[Jhonhenrygomez]

Hello, in Fabric, there is a way for incremental loading to be done without a watermark, meaning it does not use fields. When a data source such as PostgreSQL is used to identify changes "automatically, " the tools must consume the WAL so that incremental loading can be done with less manual processing. I want to know if Fabric fulfills this function because, under what you mentioned, a 200GB table (for example) would take a long time to refresh and must have a date field to validate incremental topics.

[adefwebserver]

Thanks! You are my "Go To" guy on this stuff :)

[directxrajesh]

Since there is no upsert..how do we handle updates to existing data at source.

[moeeljawad]

Hello Amit, that is wonderful thanks for sharing, at 18:42 you had mentioned that using the lookup activity is not the best practice if the table is very large, and you mentioned using a table approach, can you elaborate more on that? would having a script activity after the copy activity that will query the copied table, and get the maximum date stored in a table in the lakehouse, and then you directly lookup from that table be a possibility?

[clvc]

could you do this with files (parquet) in the lakehouse using incremental data?

[PrabhakaranManoharan]

Hi Amit. That was a great tutorial. Can we get a video on the same scenario when the data source is .csv/excel files rather than an SQL Server?

[anushav]

Hi Amit! How to work with REST API data to append the incremental data into Fabric. Do i need to reset any of the steps or do i need to follow the same procedure.

[sansophany]

Thanks for sharing. this is very helpful. I wounder is it possible to do realtime data ingestion (realtime pipeline) to lakehouse or warehouse?

[remek]

Do you happen to know if mapping data flow will be available at some point in Fabric?

[christianharrington]

Thanks great video! 🙂
I wonder about the strategy when data can be modified. So for ex. if you load purchasing document or any other document, some attributes of measure already loaded to the warehouse might change. In this case, it's not only a matter of adding new records, but also updating them. Since there is no concept of unique primary key in Fabric so far (that might change thought), I wonder how to achieve this?
That reminds me of the datawarehouse 101 old days where we needed to reverse documents that have changed, say original doc was qty 100, now you load same doc changed with qty of 90, so you needed to add a record with qty -100 and another one with qty +90.
There might ne some easier solution for sure. Any clue? 😊
(and there are deletes as well!)
Thanks!

[saikrishnanimmagadda]

Hi Amit,
I have been actively following your instructional videos on data ingestion via Data Pipeline, specifically from On-premises to Fabric. While attempting to implement the process, I consistently encounter the following error message. I am reaching out to seek your expert guidance in resolving this issue. Your insights and assistance would be greatly appreciated in helping me overcome this obstacle in my data ingestion efforts.

Thank you in advance for your time and support.

ERROR [08S01] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Socket closed. ERROR [HY000] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Can't connect to server on ERROR [01S00] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Invalid attribute in connection string: sslmode.

Kind Regards,
Sai.