What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2024)

What is Data Pipeline | How to design Data Pipeline? - ETL vs Data pipeline
#datapipeline

📢📢 Subscribe to my FREE newsletter "Normal I.T. Guy" to know more about what I am thinking, learning and doing in my career & life !!
CLICK THE LINK BELOW 👇👇👇

***Do check out our popular playlists***

1) Latest technology tutorial (2020) -

2) Google Cloud Platform Beginner Series (2020) -

Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel I.T. k Funde.

More about this video -
In this video, we will understand what is a data pipeline with the help of a real-life example. Data pipelines are designed to move data from one point to another. In this video, we will cover below topics -

1 - What is a Data Pipeline?
2 - Why you need a data pipeline?
3 - Basic design of a data pipeline
4 - Types of Data Pipeline - Batch, Streaming, Lambda architecture
5 - Advanced data pipeline design

We will also learn about various products that can be used in a data pipeline - SAP BODS, Mongo DB, Apache Kafka, Big Query, MDM, Teradata, SAP Business Objects, Tableau.

Credits & Resources -
Wikipedia

**********************FOLLOW US ON**************************
****************************************************************

******************About This Channel*************************
Friends ITkFUNDE channel wants to bring I.T related knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

[sivanagarajugamidi]

Master piece tutorial for data engineering

[nathancarranza]

Something I’ve noticed is that Indians are good teachers and give great illustrations. Good work. Greetings from the US.

[altamashjawad]

Loved this video, probably the best explanation on advanced data pipeline out there. If in your next videos, maybe create a playlist which can show each of the section of this pipeline in detail with little examples using Python or any language etc. Just an idea, brilliant work!

[ericdasse]

That was great! As a data engineer in the making, this is the first time I have understood the concept of data pipelines so clearly. Thank you very much

[MrBignate]

Please continue to create videos like these! So easy to understand. Love your visual teaching style and the examples you give.

[sy-vfjs]

And again, another easy-to-digest video. Thumbs up!

[MrBignate]

Would love to learn more about how to choose the right frameworks/technologies for data pipelines and data warehouses/lakes for differing requirements. It would be nice to see a playlist of you designing or comparing solutions for an analytic stack.

[sidr]

Thank you so much for a great and easy to understand data pipeline introduction. I love how you focus on the concepts and not jargons, as it allows for people to understand the essence of data pipeline.

[ci]

This was extremely helpful in my new job working in data. Thank you for your approachable and easy to understand teaching style!

[squareheadc]

Great intro, just what I needed. I learned the distinction between ETL and general pipe lines, and Kafka's place in the architecture.

[jananisri]

One of the best tutorials in youtube so far which gives an overview of data engineering process and that too within 10 minutes. Really appreciate your effort and time you put into making this video. Thank you so much. Please keep doing more such tutorials.

[lcsxwtian]

Simply one of the best videos on data pipeline on Youtube. Deserves so much more attention.

[aitemiryeskenov]

Wonderful explanation. Can't tell you how much I appreciate brilliant tutorials like this one.

[prabur]

This would be the Best start for the Data Engineers.. A clear precise and short pictorial representation of Data Pipeline (Basics). Best video so far I had seen.. 😊 Thanks.. Much Appreciated.. 👍

[alexanderulloaopazo]

Thank you! I had read a lot of papers about Data Pipeline, but I couldn't get the main idea. However, your video was so easy to understand!! Now I have a better picture of the complete process. Thanks again.

[jamesmcmurtry]

Great visual layout. Would love to see this applied to an ELT model with Snowflake and it's advantages/disadvantages. Possibly a suggestion on ML complementary tools like Looker and Kraken.

[mardiidking]

This topic is so complex as a beginner, but I understand this explanation so well. I didn't even have to go back in the video or rewatch it to understand. This is beautiful.

[bigglesharrumpher]

Great video - it seems while technology has advanced, the concepts of batch loads and real-time data is actually decades old. Back in early 2000's we controlled all ETL and real-time loads with Unix or DOS or SQL scripts that provided return codes for success/failure which triggered alert emails, and we had KPI's for Data quality, backing-out jobs for failed loads, and many other control systems. It just seems there are more 'out-of-the-box' software to handle these now as opposed to custom-built solutions. Great presentation!

[chimmenegaius-assor]

Great explanation for introduction to data pipelines. Thanks for clarifying the distinction between ETL and data Pipelines.

[mitchelleleeuw]

☺️I’m new in Data Engineering and man you created a clear picture of what I’ve been learning and trying to understand 🙂love this… definitely subscribing 🤩