Data Engineering with Python and AI/LLMs – Data Loading Tutorial

Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.

Course developed by Alexey Grigorev & Adrian Brudaru.

⭐️ Contents ⭐️
Alexey's part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?

Adrian's part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:15 19. Deployment with Git Actions
2:29:04 20. Deployment with Crontab
2:40:05 21. Deployment with Dagster
2:49:47 22. Deployment with Airflow
3:07:00 23. Create pipelines with LLMs: Understanding the challenge
3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38 25. Create pipelines with LLMs: Demo

🎉 Thanks to our Champion and Sponsor supporters:
👾 Drake Milly
👾 Ulises Moralez
👾 Goddard Tan
👾 David MG
👾 Matthew Springman
👾 Claudio
👾 Oscar R.
👾 jedi-or-sith
👾 Nattira Maneerat
👾 Justin Hual

--

[mrbartuss]

Thank you! This channel definitely needs more data engineering content

[darkfactsegg]

I love this course or tutorial whatever it is because I am learning DE for about 1 year on my own then realize that learning is easier when I am doing practically. Thanks and more about DE tutorials please.

[gopalakrishna]

Please upload more data engineering content especially building data pipelines in the aws. Would be highly appreciated if you can add projects on this topic

[Tenebrisuk]

This felt less like a course around data engineering and instead just a tutorial on the dlt package, which is fine, but misleading.

[johnmadsen]

A senior showed me a trick to speed up the processing .. something about piping it to dev null. It sped it up. Boss is testing it out now.

[Konko]

It’s almost like you’re reading my roadmap 😮

[FerdawsOmarkhail]

Hope everyone learns things they want ❤

[ranvijaymehta]

Thank you so much for providing data engineering contents.

[somdubey-vq]

Is this tutorial beneficial to data analysist also

[jimdaily]

Seems like this is showcasing a few different types of ingestion? Basically a few tutorials ib one video? Yeah?

[ONE_Lifə]

Import. Casoh

Remove/ storage ( auto)

[priyatiwari]

Than you for this . But it would me more helpful if we get more data engineering content please . like snowflake

[john-gnwv]

Is there an embedded function in dlt if you set data contract to send discarded_row or values to alert and store those datas somewhere ?

[LakshmanKumar-ez]

Day 3 to please post dubbed in Hindi 😢😭🙏🏽

[Un-USA]

I don’t understand this guys accent who’s explaining data ingestion