Capture Changed Data in Azure Data Factory - Handling deletions in Incremental Loads - 3 simple ways

preview_player
Показать описание
In this video, we see 3 different but simple ways to perform incremental loading on Azure Data Factory. We also see how to handle deletes in incremental loads as it is a very common scenario. Copy Activities and Mapping Data Flows on Azure Data Factory are used for the implementation of these examples. You will find the SQL Code on my GitHub account.

Follow me on social media:

00:00 - Intro
01:00 - Setting up the tables & inserting data
01:54 - Upsert using Dataflows
08:15 - Handling Deletions using Dataflows
13:30 - Testing Method 1
16:04 - Method 2 with Copy Activity and DataFlows
25:40 - Method 3 with Copy Activities and Stored Procedures
26:09 - Conclusion
  1. Apostolos Athanasiou
  2. CDC
  3. Azure Data Factory
  4. CDC Azure
  5. CDC with Data Factory
  6. CDC on Azure
Рекомендации по теме
104. CDC (change 0:24:44

104. CDC (change data capture) Resource in Azure Data Factory | #adf #azuredatafactory #datafactory

Change Data Capture 0:11:50

Change Data Capture in Azure Data Factory

39. How to 0:10:28

39. How to Capture Changed Data using Data flows in Azure Data Factory

Track and Record 0:15:33

Track and Record Data Changes with Change Data Capture (CDC) in Azure SQL | Data Exposed

Change Data Capture 0:15:53

Change Data Capture and Managed Airflow in Azure Data Factory | Azure Friday

Change Data Capture 0:58:44

Change Data Capture in Azure Data Factory

CDC (Change Data 0:12:05

CDC (Change Data Capture) Capabilities in Azure Data Factory and Azure Synapse Analytics

Change Data Capture 0:20:35

Change Data Capture in Azure Data Factory | Azure Data Factory| Data Engineer | Session 25

Day 1 | 1:01:43

Day 1 | Azure Data Engineering Training | 05-08-2024 6:30 AM | | Live class link in description

How CDC Works 0:07:52

How CDC Works in Azure SQL Server Managed Instance | Change Data Capture In Managed Instance

Azure Cosmos DB 0:06:17

Azure Cosmos DB analytical store Change Data Capture

TDPC June 2022 1:03:21

TDPC June 2022 Event: Implement change data capture in Azure using Synapse Analytics

#40. Azure Data 0:05:54

#40. Azure Data Factory - Capture Changes to the Source File Metadata without changing Data flow

SAP CDC in 0:03:12

SAP CDC in ADF

Attunity, Azure SQL 0:08:33

Attunity, Azure SQL DW, and SAP Offload with Change Data Capture and Qlik

CI/CD in Azure 0:15:18

CI/CD in Azure Data Factory: Git Integration

Real-time Change Data 0:09:56

Real-time Change Data Capture: Structured Streaming with Azure Databricks

SAP connectivity in 0:56:39

SAP connectivity in Azure Data Factory by Ulrich Christ

Loading Data into 0:30:40

Loading Data into Azure Data Lake Gen2 with Azure Data Factory v2 | Webinar

Azure Data Factory 0:34:00

Azure Data Factory October 2022 Updates

Implementing CDC with 0:35:07

Implementing CDC with change feed | Azure Cosmos DB Conf 2022

#49. Azure Data 0:17:19

#49. Azure Data Factory - Implement Upsert logic in Mapping data flow.

Azure Data Factory 0:32:53

Azure Data Factory January 2023 Updates

Move Oracle to 0:04:28

Move Oracle to Azure SQL Server in Real Time

Комментарии
Автор

Wow! This is best incremental load demo video I can find so far. Thank you!

d.b.
Автор

how do we manage delete? if source table is huge (size =200gb) how do we identify if some of the rows are deleted from it before inserting into target table? I dont want to delete the row in target table but just have a flag against it. The target table is in a DW which should have history of all data. CDC is not an option. Thanks

gauravdevgan
Автор

Can we combine the upsert and deletion in one single data flow?

AlejandraDuarte