01 Cloud Dataflow - Pub/Sub to Big Query Streaming

This video will explain how to setup a data flow job that moves data from pub/sub topic to Big Query table.

Commands used:
gcloud pubsub topics create MyTopic01
gsutil mb gs://dataengineer-01
bq mk mydataset01

Message format
{

"name" : "John",
"country" : "US"
}

[sumanthskumar]

Absolute legend! Was searching for this tutorial everywhere. Thank you so much!

[anandakumarsanthinathan]

Very useful. I wish you showed us how to create the Dataflow job also via gcloud commands. Great video, anyway.

[hamids]

Fantastic video, you covered CLI and GUI both, nice soft tone and very good language. Thank you :)

[DKC]

A very helpful video! Thank you very much for such a concise introduction to data flow

[NazarRud]

I just would like to say thanks for your learning tutorial

[aakashgohil]

im getting this error on BIgquery output table - Error: value must be of the form ".+:.+\..+"

[goodmanshawnhuang]

Great explanation. Thanks for sharing it. Well done.

[ottawabiju]

Thank you, but I am trying to export the SCC findings to BigQuery export, so created pub-sub topic/subscriptions, BigQuery dataset, and tables. PubSub does not push data to BigQuery (created table schema manually), and could not find the auto-detect schema from the configuration tool. Issue is pubsub data is not exporting to bigquery and could not figure it out.. any help would be greatly appreciated.

[itech]

hi I'm searching like this Publish rows of CSV to PubSub, then dataflow read from PubSub topic and write output to GCS if you can make a video

[suvratrai]

Thanks for the demo, , this is a great resource for anyone who is exploring the data streaming use case on GCP.

I have a question on the JSON Messages that we can process from this streaming pipeline. Can we send multiple JSON elements together to be processed with this or the system only expects a single JSON element sent on pubsub at each time? If yes, what should be the JSON structure?

Can you also let me know if this can process nested JSON's as well? How do we specify the JSON parsing logic in that case?

[tomasoon]

very helpfull tutorial, i really appreciate this!

[DannyZhang]

Great video. Thanks. Noticed on your video, you published a message with name = "Jon", then next sent a message with name "Raj". however, on your query result, Row 1 is "Raj" whereas Row 2 is "Jon", i was expecting the first message sent to be Row 1, which would be "Jon". Any thoughts?

[connect_vikas]

Need help, could you please add what permission required.

I am getting error invalid stream on failed message

[bikergangnam]

How to write 3 or 4 records, you have just entered one record or one row, can you explain if you can add more rows in same publish message

[alacrityperson]

Hello,

As pub sub is auto scalable and its having own storage then why we need to create storage bucket for pipeline job

[pranaybhaturkar]

Very helpful can also make tutorial for dataflow with golang

[richardwng]

Nice explanation, I wonder how if we create a message in pub/sub which has a different Attribute/Field from the Bigquery Table? Is it okay to do so?

the message would be something like :
{
"name" : "Andy",
"country" : "ID",
"gender" : "M"
}

while the table only contains the name and country. Thanks in advance

[onelastmarine]

the reason you select statement didnt work 11:50 was because you had a portion or your code highlighted and thats all it ran.

[mjoteyh]

how to connect the pub sub topic to another data source

[gcpchannelforbegineers]

What could be stored in the bucket ?? Like table is stored or any other is stored.

I like your explanation :) thank you. Could you plz answer above query.