Data Engineers need this lineage trick!

In this video I take you through creating data lineage within your data set. This simple approach allows you to keep full and detailed information on what has changed, where data came from, how it was processed and more right there with the data itself. No need for centralised solutions, no need for extra tooling, and no need for connectors and plugins. This is an operationally very efficient technique and quite possibly allows the storing of the most detailed information since the schema you use for your lineage is up to you. You can also use this technique to save quality information alongside the data.

0:00 - Introduction
3:25 - Demo setup
4:18 - set up demo data
5:03 - Data Factory Pipeline (ADF and CSV)
9:16 - Data Flows Pipeline (Spark and Parquet)
13:35 - View in Power BI
14:49 - Wrap-up