Building a Data Pipeline

Let's break down the common components of a big data pipeline, and how to build the overall architecture for a pipeline. And then how the tools of the most popular cloud platforms fit in, for azure, aws, and gcp.

⏯RELATED VIDEOS⏯

------------------------------------------------------------------------------

------------------------------------------------------------------------------

🎓Data courses (Not Produced by nullQueries)🎓

------------------------------------------------------------------------------

📷VIDEO GEAR📷

💻VIDEO SOFTWARE💻

------------------------------------------------------------------------------

Some of the links in this description are affiliate links and support the channel. Thanks for the support!

------------------------------------------------------------------------------
00:00 Intro
00:35 Why a pipeline?
00:55 Collection
01:23 Ingestion
01:45 Storage
02:09 Compute
02:47 Presentation
03:05 Azure
03:35 AWS
03:57 GCP

[nullQueries]

What are your favorite tools for data pipelines?

[Pjotterbolte]

Awesome video! Nice to see the tools across the different platforms

[ardavanmoinzadeh]

keep it up -- it's amazing how you explain all that in less than 5 minutes!!

[astronemir]

This beats taking 4-8 hours courses on each platform, doesn't it... Thanks a lot!

[ignacykl]

More Data Engineering concept please, this is just too good!

[vishwassawant]

@NullQueries, I really like the way you are explaining, teaching !! Examples, Presentation, Matching-Comparing Analogies. !! I used to do same way when I explain to others.

[saeed-khan-cs]

Awesome video. Thanks a lot for the really helpful contents!

[coding]

Null queries uploads a video. Instant like. Been Waiting for long since your last spark video

[ratnasambhav]

I am dying to see such videos. Really awesome video.

[svgenterprises]

how do you not have more subs?!?! great content. thank you!

[lockercoin]

very nice!
you sir, just got a new subscriber

[lentilgod]

I was under the impression that each step in the process from collection to visualization is in and of itself a data pipeline. I feel like this changes from source to source

[seancochran]

Is much care given anymore to the timing of data extraction from transactional systems? Curious if this is still a major concern for data engineering, or if the need for careful timing is mostly eliminated by modern data infrastructure. Sorry if the question is ignorant.

[christopherbronson]

I feel like that pipe on the thumbnail was on purpose to see if bad design gets more or less clicks!

[igorcotruta]

It's sad that Bi devs don't coin stack terms like web developers do (LAMP stack or MEAN stack).