Code along - build an ELT Pipeline in 1 Hour (dbt, Snowflake, Airflow)

How to build an ELT pipeline in 1 hour, using industry standard tools such as dbt, Snowflake and Airflow. This is a live coding tutorial, where I’ll walk you through the thinking process, and show you every step. We’ll cover basic data modeling techniques (fact tables, data marts), snowflake RBAC concepts, and how to orchestrate a dbt project using Airflow. Drop down in the comments section what Data Engineering topics you would like to see in the future!

Timestamps ⏰
0:00 - Intro
1:35 - Setup dbt + Snowflake
11:45 - Create source and staging tables
17:41 - Transformed models (fact tables, data marts)
20:25 - Macro functions
24:23 - Generic and singular tests
29:35 - Deploy models using Airflow

Live Coding notes 🎴

Who am I? 🙋🏻‍♂️
I'm Jay, I love making videos about travel, self-help and tech. I currently work in New York City as a data engineer, but I grew up in Malaysia and lived in the UK when I was 19. Back then, I had no idea what life was about, moving to so many places, navigating career in Tech. Today, I've learned a lot and wanna share my perspective through filmmaking.

Socials 📱

Sub Count: 7,163

[mohitupadhayay]

AN ABSOLUTE GOLDMINE OF AN INFORMATION WHICH NOT AY UDEMY OR YOUTUBE TUTOR HAS PROVIDED YET!

[GeorgeNyamao]

Thanks @jayzern. This tutorial is awesome. I will be recommending it to folks who struggle with connecting dbt with any database engine.

[twickAttack]

Hey Jay, thank you for the video. I'd be happy to see you doing more ELT pipelines and focus on your thought's process ( I can watch longer format 1-2 hours) - why you do things in that way, why is it important and whatnot; and you can throw some explainers to anything else you do and the reason behind it. I think senior DE and others with experience do things bit automatically and it takes time for the newbies to pick up on those skills. So, your thought process for doing things instead of just doing the things is priceless for anyone watching, including me. Appreciate your video, dude :)

[idxjohs]

This video is like a gold mine for building a portfolio especially for someone starting out as a Data Engineer like me!... Manny Thanks and Kudos to you!.. Love from India

[CosmicNomad]

This is such an amazing video @jayzern! The project taken was not overly complex but also not barebones and covered a lot of important stuff! Thanks for being thoughtful and including the code along link (else some of formatting issues would have bugged many newbies)!
I think you should keep creating more videos as you are a good teacher. Only suggestion I have is may be include a bit more explanation, which will help beginners even more! Kudos!

[kmonish]

I have been struggling with dbt and airflow for a long time. For some reason I could not connect the dots. Having some mixture of knowledge - I landed on this tutorial and it just glued all my scattered dots well. Thanks Jayzern!!! Really appreciate the efforts :)

[Dev_Zyoom]

honestly never knew about dbt and glad to learn it here thank you

[anikethdeshpande]

i'm new to snowflake, dbt and airflow,
this is awesome tutorial, got to learn a lot
thank you jayzern

[diaconescutiberiu]

Awesome video! I already recommended this to my entire team. Please make more like this, they are extremely helpful.
Idea for next video: dbt for Snowflake (again) but with Data Vault 2.0 modeling. I would love to see the logic behind creating dim and fact tables, how you define the stg files for creating the hubs/satellites/links.

[PhiNguyen-izgo]

This code along session (starting from scratch with environment setup, codebase structure ...) is helpful. Hope to see more examples like this. Keep up the work my man <3

[anduamlaktadesse]

so supportive and completing the project .

[rakshavishwakarma]

thank you so much it is 100% worth and useful... expect some more videos in detail.... like prod deployment through git and git interation with airflow

[nothan_nah]

Thanks for sharing this dbt tutorial! It’s definitely super hot rn and useful to learn. 🎉

[PhanNguyen_]

thank you so much for this tutorial. hope you have more videos in the future

[giovannimaia]

Please post more videos, your videos are awesome and very instructive

[projectveritas]

Great video and explanation. we need more videos from you.

[JohnS-erjh]

Thanks very much for posting this! Definately earned another subscriber/viewer

[funbadult]

Can you please post more videos like this? Really appreciate it. Helps me understand the Dbt/Snowflake/Airflow a lot

[maikerodrigo]

Great tutorial, i've learning a lot thanks!

[oludelehalleluyah]

I haven't a lot from this tutorial...

Thank you