Building ETL Pipelines Using Cloud Dataflow in GCP

This demo reads a csv file from cloud storage buckets, transform using apache beam sdk and finally load the json schema of the intended output into BigQuery.

Connect with me here:

🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY YOUTUBE CHANNEL

#gcpcloud #datafusion #bigdata #dataengineer #cloudplatform #dataflow #etl #gcpdataengineer #bigquery #cloudstorage

[venkatvlogs]

too hurry not able to understand it as you are switching tabs and doing all the things and not mentioning where you are writing the code. The course should be designed so that even beginner should be able to understand it. please make a pin to pin point to point explanation video so that everyone can understand it. Thanks in advance ❤

[rrafaelpaz]

Very nice mate! Very well explained! Cheers from Brazil brotha!

[chaithuchinna]

is there any course available sir to learn gcp ?if so pls help me provide the details

[pournimaambikar]

I am getting below error while trying to run dataflow job:
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
on both cloud sdk and cloud shell, wheras apache_beam is installed

[ashwinjoshi]

Thanks for the video. One question - in case the source is oracle on premise and sink is BigQuery then what changes are required to do ?

[student_voice]

And also, one more request, when you using a gcp Service, also explain required its access privilege for a user

[ashishvats]

Great video, i want to take input from JDBC connection a table and load to bigquery… could you please share any document related to this, to how take table as an input from JDBC and load to bigquery

[sumitdwivedi]

can you create this pipeline and do transformations within gcp dataflow itself?

[sanketgurnalkar]

How to give runtime parameters? can you give the code

[student_voice]

Can you make a video on CI/CD for from oracle to bigquery using tools like jenkins bitbucket sonarqube checkmarks, airflow Composer..
If u can, this will be very helpful.. 🤝

[tommedcouk]

Dataflow isn’t the most widely used component in the Google Cloud Platform. Even if you Google this question, the sensible response is Compute Engine because it runs under pretty much all the other services, but also because a lot of companies do a lift and shift to cloud before integrating with the other services. You claim this twice at the beginning of the video, but it’s incorrect

[AnantPradhan-ym]

Couldn't understand. Complicated...

[pm]

very confusing you keep jumping from 1 screen to