What is this delta lake thing?

Показать описание

You may be using a lake for your data and it may just be regular parquet files. In this video, Stijn joins us to explain why you should be using a delta lake instead and how this works in Azure Synapse Analytics.

Connect with Stijn

What is Delta Lake

Delta Lake Documentation

*******************

Want to take your Power BI skills to the next level? We have training courses available to help you with your journey.

*******************
LET'S CONNECT!
*******************

***Gear***

#AzureSynapse #DeltaLake #GuyInACube

Рекомендации по теме

Комментарии

This video is gold, makes it easier to understand spark and delta lake - kudos!

cantTouch

Again, another great video from the great series (Azure synapse analytics)
Thanks a lot guys(in the cube), you are amazing

mohamedtarek-ghfr

How do you handle change to the source system in a Delta lake? For example: when a source table adds 3 columns and drops two?

dancrowell

Interesting. What is the benefit of using this vs creating incremental loading within your merge statements? Are there more costs associated with using a delta lake? Additionally, will this pick-up changes from my source?

ChronicSurfer

Cool video. Those DROP TABLE IF EXISTS and DROP DATABASE IF EXISTS are precautions so we don't run into an error when we replace what's there, right?

matthiasg.

@Guy in a cube - Can we do this same thing in ADF - Mapping dataflow?

nagoorpashashaik

To be clear, bronze layer (parquet, json, csv, ...etc), silver layer (delta lake, iceberg or hudi - one table open format) and gold layer (SQL Query - Views) to server data users ? is it correct?

ufiepte

Hello! excellent video! It is recommended in the first bronze layer to save the data in parquet and in the following two in delta? thank you

maxirojo

Can an API hosted on an App service in anyway fetch Delta tables data ? thanks

sid

Great video series for getting started with the topic. Probably the video is already in production but as a follow up to the series I can imagine it could be interesting to see how powerful the functionality of delta is. What exactly does the time travel feature look like. For me it was impressive to see how granular you can jump back in time and roll back changes to rows but also structural changes to a table. If we want to look at it more from an ETL perspective, maybe a look at the change data feed would be interesting.

Regardless of how you continue this series I am very excited because your hands-on way of approaching these things takes the hurdle out of many to begin their journey.

PCGHigh

Do you know what is the difference between lake databases and delta lake project? Both seem to have roughly the same functionality - I can use Spark to do ETL tasks - and then use spark pools as well as serverless sql pools to query data.

googlegoogle

Hello, thanks for putting out great content and useful videos.

Delta is certainly cool, however, after having a deeper look: Delta time travel does not seem to be a replacement for a proper Type2 SCD modelled data, since:
- there is a limited data retention for the delta log (30 days), it can be extended of course
- you can't leverage that time travel when using Serverless SQL Pool (which is how I'd expose Delta tables to Power BI)
- or have I missed something obvious?

Furthermore - the SQL / pySpark interoperability works to an extent, for example Synapse Spark SQL doesn't support SQL based time travel (SELECT * FROM TABLE AS OF VERSOIN N) - this has to be done via pySpark. On the bright side - pySpark is not that hard to pick up, takes getting used to, but it's quite powerful :)

Now only add support for Delta for the Workspace-created Lake Database! :)

Cheers

radekou

Previously I would not have perquet files, previously I would have a SQL-Server. What problem does a delta lake solve compared to just using a SQL-Server?

martinbubenheimer