What is Dataflow Prime?

We’re excited to announce Dataflow Prime, the next generation of Dataflow: Google Cloud’s truly unified batch and streaming data processing product. Dataflow Prime has two new features: vertical auto scaling and right fitting, which provide automatic fine tuning of memory and stage specific resource allocation. Plus, a new pricing model to simplify your billing.

Chapters:
0:00 - Intro
1:02 - Understanding Dataflow
2:02 - Vertical autoscaling
2:52 - Example of vertical autoscaling
3:44 - Horizontal autoscaling
4:30 - Right fitting
6:00 - Example of right fitting
7:06 - Recap of new features
7:41 - Simplified pricing
9:04 - Wrap up

#GoogleCloudTech #DataflowPrime

[phelipe]

Is DataFlow Prime already available in airflow operators like classic data flow to create jobs more easily?

[hashiromer]

Would it be possible in future to get rid of vertical/horizontal scaling and dealing with worker nodes altogether by doing best bin packing?

I am imagining a system where I just describe the workload and it takes care of finding the best way to execute it on hardware. It may have to find n nodes, each with some capacity C at time t while optimizing for either time/cost or a supplied tradeoff between two.

[shirleyzhou]

This is not AOPS because it’s a different channel and different person 😮 and I was supposed to be watching AOPS!

[spectrum]

Is it dataflow with same day delivery 😂