Microsoft Fabric: Incremental ETL for Warehouse using Data Pipeline, SQL Procedure

This video will discover the ways to run Incremental ETL(Data Load) using Data Pipeline and SQL Procedure in Warehouse. Warehouse Support read and write SQL to SQL procedure has been used. In this Video I have not used Data Flow Gen 2

00:00 Overview of Incremental ETL
03:00 Create Warehouse and Load Data
08:30 Create Data Pipeline for Incremental Data Load
19:30 Load Data Using SQL procedure and Pipeline
23:30 Load Data Incremental Data using pipeline again
27:30 Analyze Data in Power BI

CREATE TABLE [dbo].[incr_info]
(
[tablename1] [varchar](4000) NULL,
[collectdate] [datetime2](6) NULL
)

insert into [dbo].[incr_info] values('sales', '2018-01-01');

(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;

(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;

AS
BEGIN

order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,
discount_percent
)
select A.* from
(select order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,

where tablename1 = 'sales';

end

🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.

With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.

Don't forget to like, comment, and subscribe for more great content!
▶️Data:

---------
▶️Follow me on:
---------
▶️My Other Videos:

-~-~~-~~~-~~-~-
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
-~-~~-~~~-~~-~-

[shafa]

By far the best series on Fabric on YouTube. Currently I am consuming anything and everything available on Fabric on YouTube . Your contents are the best. Keep up the good work.

[dyxuujk]

Hi Amit, Good Morning!, I have to convert the existing SQL server stored procedure into fabric environment, In my stored procedures there are CURSOR commands but fabric doesn't support CURSOR commands, in this case how do I proceed, is there any alternative.

[ljfkdgh]

Hello sir,
I wanted to ask that:-
How can we delete a table residing in Fabric warehouse from Pipeline.

It allows to delete from a Lakehouse, but not from warehouse.

Can you please assist here

[supreetkaur]

I am trying to create table in fabric warehouse but seems like I have no access to create how to fix it.

[notoriousft]

How did you create on-prem Postgres connection? Cause currently data pipelines in Fabric don't let you connect via on prem data gateways. Are you using some cloud hosted version such as elephantsql?