Data Engineering on Databricks Demo

In this demo, we give you a first look at Delta Live Tables, a cloud service that makes reliable ETL – extract, transform and load capabilities – easy on Delta Lake. It helps data engineering teams simplify ETL development with a simple UI and declarative tooling, improve data reliability through defined data quality rules and bad data monitoring, and scale operations with deep visibility through an event log.

Connect with us:

Video chapters:
00:00 Intro to Delta Live Tables
00:45 Data ingestion with Delta Live Tables
02:15 Data transformation with Delta Live Tables
03:37 Data quality validation with Delta Live Tables
06:00 Working with Delta Live Tables data pipelines
07:37 Databricks Jobs integration with Delta Live Tables

[PayoBeg]

Amazing video. Not only informative, but it also boosted my interest for data engineering greatly! Thanks!

[neelred]

Can you make the notebooks and datasets used in this demo available to download ?

[singhprabhjot]

When I try this live table creation thawing this error :-
This Delta Live Tables query is syntactically valid, but you must create a pipeline in order to define and populate your table.

[kurtmaile]

Yes can we get a link to the notebook please? Great presentation

[neelred]

I assume that for Continuous pipeline, if the total run time is for example 15mins, pipeline will start from the beginning at the end of the previous run. It would effectively be similar to for example job scheduled at 15mins interval.. Can you please confirm this?

[rienadetrof]

how to add new data in existing live table? Is it automatically done? For example if I add an updated file on a blob storage how it is updated in my tables?

[sreelakshmisandeep]

very informative :) could you please share the notebook ?

[RajanieshKaushikk]

The biggest issue with Databricks Demo is they are not producible if the customer wants to try it.Also code is not available to play around with it.

[oiwelder]

Olá Databricks, poderia expandir o vídeos e documentação para o idioma em português. Engenharia e ciência de dados tem crescido de forma exponencial no Brasil.