Azure Synapse Analytics | Data Ingestion Patterns, Polybase, and Copy Command

Having right data ingestion framework and patterns are important considerations to have consistent data ready for analytics in timely manner, especially when you are dealing with large data volume and expect your data to grow every day.

In this video, you will learn about different data ingestion patterns and options available with Azure Synapse Analytics - dedicated SQL pool.

00:00:00 Introduction
00:00:28 ELT (Extract Transform and Load) vs ELT (Extract Load and Transform)
00:01:57 Single Gated Bulk Data Load
00:05:06 Parallel Data Load
00:07:01 Parallel Data Load - Polybase
00:09:42 Data Import - with CTAS
00:11:52 Data Export - with CETAS
00:14:58 Parallel Data Load - Copy Command
00:19:25 Demo - Bulk Data Load with bcp
00:26:55 Demo - Bulk Data Load with Polybase
00:53:07 Demo - Data Import (CTAS) vs. Data Export (CETAS)
00:57:32 Demo - Bulk Data Load with Copy Command
01:11:49 Demo - Incremental Data Refresh
01:17:29 Demo - Monitor Data Load Operations

Thank you for watching, in my next video, I am going to talk in detail about multi-layer security architecture available with Azure Synapse and how you can build modern data warehouse with confidence in Azure Synapse. Stay tuned.

GitHub Repo to download deck and script used in the video:

Data loading strategies for dedicated SQL pool in Azure Synapse Analytics

Design a PolyBase data loading strategy for dedicated SQL pool in Azure Synapse Analytics

CREATE EXTERNAL TABLE

CETAS with Synapse SQL

Bulk load data using the COPY statement

What is the file splitting guidance for the COPY command loading CSV files?

How to maximize COPY load throughput with file splits

bcp Utility

[daizhang]

this is very well organized content, defnitely on the same level as microsoft traning

[sagarvarma]

Thanks for creating the entire series on azure synapse E2E with demo class. It becomes much easy to understand the point if we can imagine, and your video content made me to imagine those concept which can help me remember forever.
Could you please start preparing video on other azure services as well like stream analytics etc essential for data engineering

[jbrajgor]

Hi Arshad, A well thought and put together video. Compared to other folks just touching the surface - your expertise and guidance for real-world use - is really helpful to many. Please continue to create additional Videos related to Synapse & Analytics in general. Thank you.

[VK-lnvk]

wondeful video as usual.Thank you for sharing your knowledge with us.

[Haribabu-zjhd]

Thank you so much for your detailed explanation. Hoping more such videos on the latest features on synapse.

[praveennaregal]

Hi Arshad, very well explained. Looking forward for your more videos on Synapse. Thank you.

[gvgnaidu]

Excellent content with Realtime scenarios 😊

[ravitutika]

Thanks for your explanation in details, we want more videos from your end

[mohanmane]

Hi Arshad, your videos are really to the details & explain all concepts in depth. Hope to see many more videos :)

[gallardorivilla]

great video! thanks for share your git repo.

[dataengineerazure]

Hello Sir Ali , Why was external data source not created before external file format? thank you in advance