Delta Live Tables Demo: Modern software engineering for ETL processing

Watch this demo to learn how to use Databricks Delta Live Tables to build a declarative ETL pipeline for batch and streaming data with SQL.

Delta Live Tables (DLT) is the first ETL framework that uses a simple declarative approach to building reliable data pipelines and automatically manages your infrastructure at scale so data analysts and engineers can spend less time on tooling and focus on getting value from data.

Connect with us:

[marcinsiara]

Great video, easy to understand, good overview for the Databricks beginners. Thanks!

[amansehgal]

This is great. Can you share notebook for querying the transaction log and presenting it in reDash?

[jolettin]

Do you have a link to show how the queries work for monitoring the data quality. Thanks

[joegenshlea]

Thank you for this video. I'm a little confused about what the "data.stations' refers to? Is it an array in the source json?

[ozland]

Thank you for the explanation! Can please someone confirm that the files can only be read from FileStore and not blob storage by this approach (SQL)? Tried reading it from blob storage and cannot access using cloud_files() function. I guess it requires pyspark to access blob storage and create delta live tables.

[gekodragon]

Good video thank you. Quick question, is the DLT lineage also auotmatically avaialable and visisble in Unity Catalog?

[willf]

Nice demo, but why does the comment for the "cleaned_station_status" table say "partitioned by station_id" when the code actually uses the last_updated_date column? You should update the comment in that notebook. :-)

[sid]

Can an API hosted on an App service in anyway fetch Delta live tables data ? thanks

[ibozhu]

It’s been in gated preview for too long, when will it be made GA?

[mohammedsafiahmed]

am I missing something or does the video really doesnt show how he got all those files in the data lake in the first place?

[nontapatsumalnop]

Anyone knows how to create a live dashboard like this in databricks?