What are some common data pipeline design patterns? What is a DAG ? | ETL vs ELT vs CDC (2022)

What are some common data pipeline design patterns? What is a DAG ? | ETL vs ELT vs CDC (2022)
#datapipeline #designpattern #et# #elt #cdc

1:01 - Data pipeline components
4:10 - ETL design pattern (Extract, Transform & Load)
7:15 - ELT design pattern (Extract, Load & Transform)
10:37 - CDC design pattern (Change Data Capture)
14:22 - EtLT design pattern (Extract transform Load & Transform)

Hi Friends, I am Anshul Tiwari, and welcome to your youtube channel "IT k Funde" where we make I.T. interesting for everyone (Tech or No-Tech).

***Do check out our popular playlists***

3) Google Cloud Platform Beginner Series -

4) Latest technology tutorial (2021) -

More about this video -

This video is a follow-up video that talks about some basic data pipeline design patterns that are used in data warehousing or data lake solutions. We will learn what is DAG (Directed Acyclic Graph) and its core components.

Then we will move on to our 3 primary design patterns and an additional bonus sub-pattern. Below are the topics we will cover in this video.

1 - Data pipeline components
2 - ETL design pattern (Extract, Transform & Load)
3 - ELT design pattern (Extract, Load & Transform)
4 - CDC design pattern (Change Data Capture)
5 - EtLT design pattern (Extract transform Load & Transform)

PLEASE WATCH OTHER VIDEOS FROM THE POPULAR PLAYLISTS GIVEN BELOW. EVERY SINGLE LIKE, COMMENT AND SHARE MEANS THE WORLD TO ME!

#itkfunde #keeplearning #keepsharing #keephustling

Credits & Resources -
Research - wikipedia

***Social Channels***

🚀🚀LAUNCHING MY 1st EVER ONLINE COURSE 🚀🚀

"Cloud 101: AWS for Dummies - Your 1st Date with Cloud !"

✨Your first steps towards a Digital Cloud Career☁️✨

Highlights:

✅ No Pre-requisites, No Coding needed
✅ Course Starts on 11th September 2023
✅ Career Boost: Unlock new career opportunities by mastering cloud basics and AWS fundamentals.
✅ Step-by-Step Demos: Follow along with our easy-to-follow demos that walk you through key concepts.
✅ Live QnA session and career guidance session with me
✅ 2 years Access to the course
✅7-Day Money-Back Guarantee: I know you'll love the course. If for any reason you're not satisfied within the first 7 days from course launch, we offer a full refund.
✅ AWS Cloud Practioner Certification Guidance
✅ Bonus: Invitation to join my special Telegram community for Lifetime
✅ Bonus: After the course, stay personally connected to me for career guidance

With a 7-day money-back guarantee, you can enroll with confidence. Don't miss out on this opportunity to learn and grow in the cloud industry.

Enroll Now and take YOUR First Steps towards a Digital Cloud Career !!
Hurry!!

***About This Channel***
Friends ITkFUNDE channel wants to bring I.T related knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

[MasQred]

While extraction of data from operational database. Won't that affect the operetional databases performance. How to extract data without affecting it.

[patmclaughlin]

Love this, man! Our engineers made it so hard to understand what DAG was. I thought I was not smart enough, but now I know they were either deliberately making it hard, or maybe they didn’t understand it themselves.

[andygarnet]

Thanks man! Your explanations so clear and straight fwd. For years I spoke to do many engineers who would over -complicate pattern concepts or straight low ball the documentation to cover themselves when the pipelines blow up and impact the business. Keep up with these great videos!

[AamirAzizYouTube]

Thanks so much, sir! This topic was a nightmare for me, you made it so simple to grasp. Keep up the good work!

[daves]

Perfect. Full respect to your kindness and sharing of your knowledge

[bajicdusko]

It always amazes me how we can have knowledge like this one click away! Fantastic content, keep up with good work.

[metaocloudstudio]

Good point, also the pros of using ELT over cons of ETL is creating normalizing tables and real-time materialized views

[deeptihazari]

How this Amazing channel was hidden till now ..this is called Quality content delivery 👍

[wendypark]

I learned a lot network and data pipeline knowledge from you. It''s really hard to learn these from a book. Thanks a lot!

[sultanqureshi]

Though its not exactly related to my current profile but its make me happy to learn more about the whole software industry from core and you are best at making this understand by making it simple. Understood the 4(ETL, ELT, ETLT, CDC) data pipeline at once.
Video was not long at all
Thanks

[raghurajsawant]

You are doing a fantastic job. Love your videos.

[mohit.srivastava]

both this and the previous connected video explained the concept really well. thanks!!

[francksgenlecroyant]

perfect video about Data Pipelines 👌, thanks!

[SeanRomberg]

Thanks for the share - you have helped me better understand the pipeline automation software that delivers orchestration, ingestion, transformation, and activation all in one. This makes sense now.

[altamashjawad]

Thank you so much, very nice and comprehensive video!

[vaidyanathashankar]

Fantastic explanation, thanks for the wonderful session.

[UTUBDZ]

Great content, thank you very much sir !

[mayurarun]

This is such a gem video. This would help me so much. Great work.

[DJEYkanjaria]

Great Video, Simple and detailed explanation

[ASHighlights]

Very helpful sir your videos converts my nervousness into confidence !!