Automating Data Loading from Google Cloud Storage to BigQuery using Cloud Function and Dataflow

Looking to get in touch?
In this video, I showcase the process of automating the transfer of data from Google Cloud Storage to BigQuery. The automation is achieved through the utilization of Cloud Functions, demonstrating step-by-step how this setup streamlines and orchestrates the data loading process seamlessly between these Google Cloud services.

Associate Cloud Engineer -Complete Free Course

Google Cloud Data Engineer Certification Course

Google Cloud Platform(GCP) Tutorials

Generative AI

Getting Started with Duet AI

Google Cloud Projects

Python For GCP

Terraform Tutorials

Linkedin

Medium Blog

Github
Source Code

#gcs #googlecloud #cloudstorage #bigquery #datapipeline #automation #cloudfunction

[abhaybulbule]

Proud to see that among of us ( software engineer) achieving their goals by dedication and commitment
Great congratulations whatever you achieved in your life

[alanguev]

You're the best, brother. The information was really helpful, I appreciate it a lot. Greetings from Argentina, Buenos Aires.

[Ranjana_DE]

I have been looking for this content for a long time, thank god I found this, very useful

[amritapattnaik]

i loved all your videos.Keep posting 😇🙂🙃

[faroozrimaaz]

Your videos are informative..keep going

[noolusireesha]

Sir, i have done the same process as you have menctioned in the video ....i'm getting error "java.lang.RuntimeException: Failed to serialize json to table row" . Could you please reply me with the solution.

[python_code]

Can we add this project in resume as a mini-project

[hunterajones]

did the schema originally fail since the headers would not be an integer? With headers removed the original scheme would work right? Also, is there a way to automate header row removal? I need to daily auto add a CSV like this but it will always have a header row needing removed. Thanks for the video!!

[earthlydope]

There's a catch here, we need to create BQ-Table-Schema and UDF.js file everytime before uploading a new flat file into the system.

[nitinhbk]

Could you please let me know what was the COST shown in GCP for this activity?

[zzzmd]

Hi, Thanks for the great informative video. can you explain the flow if the data source is from a Rest API. Can we have a dataflow configured to extract from a Rest API to big query with dataflow without having cloud functions or Apache beam scripts involved? Thanks a lot in advance..

[zjyxrku]

Very good Video.
Where can I get more cloud function templates ??

[arerahul]

Insightful video. Just a question - Cant we write the data load job in cloud functions, rather than using dataflow ? Also how do we create a delete job - data is deleted whenever the file is deleted from GCS

[vinnakollurakesh]

Hi sir can you help me to pull the data from Kinaxis rapid response API to GCS, any related documentation or videos will be helpful, thanks

[subhashs]

Which location was template path in Python file?

[srikarfarmacy]

thank you for the video, i have one doubt, if my csv file have header then do i need to have JSON code for schema?

[mulshiwaters]

This is what exactly I need however instead of Trigger I would like to use Schedular with certain time interval like Daily Weekly . How can I achieve this.. Cloud COmposer ?Workflow ? Schedular

[ayush_]

Hello sir,
I watched your lot of videos related to function, dataflow
I have one question
As a GCP Data engineer who is responsible for writing code for dataflow or for data fusion?

[hxhqstr]

You could do away with dataflow here. A simple python job using load_table_from_uri with auto schema detect enabled from trigger function would do this work.

[pramodasarath]

Do we have to select csv file from storage to bigquery Or text file