Spark Data Engineering Patterns – Shortcuts and external tables

Fabric Espresso Presents: "Spark Data Engineering Patterns – Shortcuts and external tables"

📌 About This Episode:

In today's data-driven world, mastering Spark and Big Data technologies is no longer optional—it's essential. That's why we've invited Daniel Coelho to share his invaluable insights. Daniel is driving Delta Lake at Microsoft Fabric and Azure Synapse Analytics. He's laser-focused on Data Engineering experiences using Spark and Big Data technologies. His work enables BI Analysts, DBAs, Data Engineers, and Data Scientists to shape their data effectively and build incredible solutions.

In this episode, we'll dive deep into:

📊 The Difference Between Managed Tables, External Tables, and Views

📊 Why It's Worth Using Delta Format

🛠️ Shortcuts for Optimizing Your Spark Workflows

🗄️ How to Leverage External Tables for Better Data Management



✅ PS. Demos starts from 08:28! :)


🎙 Meet the Speakers:

👤 Guest from Product Group: Daniel Coelho, Principal Product Manager



👤 Host: Estera Kot: Senior Product Manager at Microsoft

[camilamribeiro]

Good to know!!! Thanks for sharing with us Daniel!!!

[alacostabr]

Daniel you absolutely amazing! Thank you so much for clarifying and all, all learners, create capabilities to better understand concepts correctly!

[azizquazi]

It is always good to provide the source file locations and the ipynb files to follow the videos. I appreciate your thoughts on this. Thanks.

[DanielWeikert]

Share more insights about the cost attached to shortcuts. This might be very relevant for pulling data out of e.g. S3
br

[effrey]

Any place where you Can see the views that has been created in the lakehouse item?

[StanleyCruvinel]

Great Dani Bunny! I’m looking for about Shortcuts tables and how create versions or branches from a data lake. Updates, Inserts only on my data branch. Closer what ‘lakefs’ do.