Tech Chat | Slowly Changing Dimensions (SCD) Type 2

preview_player
Показать описание
We will discuss a popular online analytics processing (OLAP) fundamental - slowly changing dimensions (SCD) - specifically Type-2. As we have discussed in various other Delta Lake tech talks, the reliability brought to data lakes by Delta Lake has brought a resurgence of many of the data warehousing fundamentals such as Change Data Capture in data lakes. Type 2 SCD within data warehousing allows you to keep track of both the history and current data over time. We will discuss how to apply these concepts to your data lake within the context of the market segmentation of a climbing eCommerce site.

Speaker:
Douglas Moore, Solution Architect

  1. Databricks
  2. data warehouse
  3. data warehousing
  4. data lake
  5. delta lake
  6. data lakehouse
Рекомендации по теме
Tech Chat | 1:00:50

Tech Chat | Slowly Changing Dimensions (SCD) Type 2

SCD: Slowly changing 0:25:43

SCD: Slowly changing dimensions explained with real examples

How to Handle 0:03:43

How to Handle Slowly Changing Dimensions

Working with Slowly 0:13:31

Working with Slowly Changing Dimensions in Power BI

Slowly Changing Dimensions 0:08:15

Slowly Changing Dimensions For Data Engineers

Understand Slowly Changing 0:23:21

Understand Slowly Changing Dimensions

Ch.02-23 Dimension Types: 0:13:17

Ch.02-23 Dimension Types: Slowly changing dimension SCD 0,1,2,3,4 | Data Modeling | DWH Architecture

IICS | Slowly 0:06:17

IICS | Slowly Changing Dimension 2 - Part 1 | #informatica

iPhone 16/16 Pro 0:35:14

iPhone 16/16 Pro 25 Settings You NEED to Change Immediately!

What is SCD 0:10:46

What is SCD / Slowly Changing Dimension | Data Engineering Tutorial | Data Engineering Concepts

Slowly Changing Dimensions(SCD) 0:27:10

Slowly Changing Dimensions(SCD) Types with Real time examples

What the French 0:07:33

What the French Toast is a Slowly Changing Dimension???

Historisierung von Stammdaten 0:07:04

Historisierung von Stammdaten Slowly Changing Dimension

ODI SCD Type 0:14:02

ODI SCD Type 2 Implementation step by step process (Slowly Changing Dimension)

How to use 0:10:54

How to use dbt Snapshots to track data history

Build Slowly Changing 0:14:07

Build Slowly Changing Dimensions Type 2 (SCD2) with Apache Spark and Apache Hudi | Hands on Labs

Rapidly Changing Dimensions 0:05:08

Rapidly Changing Dimensions | Fast Changing Dimensions

ODI SCD TYPE 0:04:14

ODI SCD TYPE 1 MAPPING

61. Databricks | 0:20:03

61. Databricks | Pyspark | Delta Lake : Slowly Changing Dimension (SCD Type2)

Ilissa Ocko: The 0:09:58

Ilissa Ocko: The fastest way to slow climate change now | TED Countdown

Spark SQL for 0:29:57

Spark SQL for Data Engineering 16: What is slowly changing dimension Type 2 and Type 3 #sparksql

Slowly Changing Dimensions 0:35:18

Slowly Changing Dimensions (SCD) Overview and Type 2 Management Tutorial

ADF MDF Training 1:01:23

ADF MDF Training Module 5: Slowly Changing Dimension

Would you do 0:00:37

Would you do this in a presentation? 😂

Комментарии
Автор

Great material again....another session on generating the surrogate keys would be great. I have seen folks adding dummy members in dimensions and initially have them set the record to invisible and just update the values as new members come in...ain't pretty but it works...

funwithazure
Автор

The link to the notebook bring to this youtube video. Can we get the notebook of this demo??

dipeshvora
Автор

Could you please share notebook you used

prasanthperumalla
Автор

can we have a some logic where we do not have to use this mergekey as extra column, as I also want schema evolution along with scd2, but using this approach with schema evolution I am also getting the mergekey column in the output, which I don't want, so is there any way?

omkarshirsat
Автор

Is there any performance difference between implementing SCD2 using Merge INTO and incorporating SCD2 by UNIONing changed, unchanged and new dataset then insert that into the target by INSERT OVERWRITE TABLE statement?

debanjanray
Автор

So we have history handled records in the Gold zone which has data in dimensional model. What about the Silver zone. Do we have historical records over there as well or do we only keep the current records from the source? What if I had to do offload reporting on Silver zone and that involves historical reporting as well..? I am offloading reports on silver zone because it has table structure very similar to that of source systems and the currently running reports on the source system tables can be offloaded here easily rather than on the Gold Zone.

omairkhan
Автор

Merge into performance to be watched closely and partition pruning strategy is not clear
Could you please cover partitioning techniques in this context ?

ahmedbedhiaf
Автор

This link is for the folder, there is a README and the notebook is the .html file. Enjoy.

dmoore