Chapter #9 - How to design data pipeline on gcp (Google Cloud Platform) ?

Chapter #9 - Designing data pipeline solution on GCP
#datapipeline #googlecloud #gcp

***Google Cloud Platform Beginner Series***

***Networking Basics Playlist***

*** Other Popular Playlists***

Latest technology tutorial (2020) -

Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel IT k Funde.

More about this video -
In this video, we will learn something different, instead of discussing any specific product on google cloud platform, we will learn how to design a data pipeline solution on google cloud platform.

This will help us in 2 ways -

1. This will helps us improve our design thinking
2. We will get an overview of all google cloud products that can be used in developing a data pipeline.

We will start with understanding a typical on premise data pipeline and enterprise data warehouse (edw) design. We will then take this same solution and try to implement it on google cloud platform.

On-Prem solution design contains below products -
#Streaming data - Apache Kafka
#ETL - Informatica, IBM Datastage, SAP Data Services
#EDW - Teradata
#Dashbaords & reporting - Tableau and SAP Business Objects

GCP cloud design contains below products -
#Streaming data - Cloud PubSub
#ETL - Cloud Data Fusion
#Storage - Google Cloud Storage
#EDW - Google Big Query
#Dashboards - Looker, Google Data Studio
#ML - AutoML
#Metadata Management - Data Catalog

PLEASE WATCH ALL THE VIDEOS IN ANY PLAYLIST FOR BETTER UNDERSTANDING !!

Credits & Resources for further study -
GCP resources documentation for detailed study -

***Social Links***

***About This Channel***
Friends ITkFUNDE channel wants to bring I.T. industry knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

[shubhamsadhuka]

This is very informative. Clean and clear of complex Jargons. Coming from a non-IT background I felt this video was very informative and descriptive. Thanks!

[metaocloudstudio]

As a Data Engineer who worked on Cloud solutions for 7 years, I have never seen such a correct and clean solution. I am interested to see the same solution on AWS as well.

[sanjayroberts]

Incredibly helpful -- interviewing for senior level data engineering positions and many have data pipeline design sections. This is perfect for that

[antonioflores]

I'm trying to build a data lake for an undergrad cloud class and honestly I was feeling pretty hopeless but after watching your video I feel way more prepared to tackle this!!!! Thank you!!!

[kaushabvithani]

Siddhi baat no bakwass perfect way / An Eye opener to a beginner / wana Pursue GCP

[AmeyChittar]

Hi. So glad for posting this. I was going through the coursera course for this and I never understood the big picture. It's so clear now. Thank you so much.

[hemanth]

Very informative to the beginners of cloud but helped with on-prem solutions, thanks tons!!!

[ankursahu]

This is one of the great video to make me understand how Project architecture is there in GCP, I would request to please create a series to build end to end demo project in GCP utilizing all these services.

[prithvinoojibail]

I am glad you have tutorials in English (since I don't speak or understand Hindi). Thanks for this video!

[gauravjha]

Very nice for enlightning with so many information

[jackleung]

really enjoy watching your video, very clear and well-explained.

[pankajmanik]

This video is too good !! So many topics covered crisply in 1 video ... Thank you very much Sir :)

[chehz]

Thank you very much for all your hard work and the nice lecture.

[LeonidAndrianov]

Thank you, mate! I like your optimism as well as the explanation.

[joannwatu]

Thank you for sharing. Your explanation of the data lake structure for both on-prem and cloud solutions was very clear and easy to follow.

[anandakumarsanthinathan]

Absolutely, absolutely wonderful and insightful presentation. Thank you very much.

Just an observation. With BigQuery as your data warehouse, companies need not go for datamarts any more. BigQuery being a scalable solution, personnel from the various departments such as Sales, HR, Marketing, Finance can run their queries against a single database and still get results back without compromising on performance. Security can be implemented at the user/department level.

[siddeshsharma]

Really Great video, been looking for such explaination

[Poornima_life]

It’s perfect explanation….you are genius man

[swarupdeshpande]

Thanks. it was good presentation. To the point.

[RizalIsmail]

Really great video. So humble you are and you give good first impressions at 1:25