Pub/Sub tips and tricks

Pub/Sub is an asynchronous messaging service that can help you easily run serverless applications. However, there are some common issues that one may run into when using Pub/Sub. In this episode of Serverless Expeditions, we run through some common Pub/Sub issues that our fictitious ride hailing app has ran into - such as handling backlogged messages, unordered messages, and much more. Watch to learn some useful tips about Pub/Sub for your serverless applications!

Timestamps:
0:00 - Intro
1:38 - Using dead letter queues to deal with malformed messages
5:55 - Using message ordering to prevent bugs
9:20 - Replaying past messages to test new versions of your application

#ServerlessExpeditions #ServerlessExpeditionsExtended

product: Cloud Pub/Sub; fullname: Tianzi Cai, Martin Omander;

[laskdkmgful]

Ordering is important. Might try to share more in detail next time. Thx. 👍

[AntonioCachuanAlipazaga]

Excellent tips! Please, a Dataflow tips and tricks would be aweosome and a video about real time analytics (Pub/Sub + Dataflow + Tensorflow)

[aaronbelchamber]

A great example used, very clear! Nice job!

[pinkanrou]

Excellent one!!! Thanks @Tianzi :)

I tried the dead letter topic one but with mixed luck :(

It worked when I tested without any subscriber . I pushed a message and checked the delivery attempt after 5 times it pushed the message to dead letter one.

When I used dataflow as a subscriber and intentionally threw exception from subscriber code it did not work. The message is not getting acknowledged as I can see from Unacked message count, but it never reached to dead letter topic. So the error count keeps on increasing in dataflow.I have not written any code inside subscriber to push the message to dead letter as I think It should be pushed automatically. Am I missing anything ?

[JosephLust]

Great presentation, Martin & Wes!!

[robinthomas]

Whats the behavior when publisher sets message ordering and publishes messages with an ordering key and there are multiple subscribers listening to the same topic and only one of them is interested in ordered messages.

[amoslam-wy]

for the message ordering, what if request a ride fails in dispatcher? since message ordering cant be supported via dead letter and if request ride + cancel ride is pulled together in the same pullResponse, the subscriber can process cancel ride as well even if request ride fails.

[GabrielAmyot-qe]

Thank for the tips. Love this format. Thanks

[PatrickSteil]

I tested code running on Cloud Run to add 100 messages to a queue and it took .04s per message. That is 40ms per message add. The message was only like 32 bytes.

Does this sound like it is too slow?

Any way to speed this up?

[ganeshsonawane]

Excellent idea. Got more insights about pub/sub through this.

[eyesopen]

What are the calls between the Dispatcher and Fleet manager (get-available_drivers, send-available-drivers)? Are these also pub/sub? How are these communicating?

[alexpalau]

Really nice tips! Ordering messages feature will help a lot of people and projects.
What will happen if order is enabled and dead lettering also? Will good messages still be sended in order eventhough some of them are missing between the good ones?

[raihankhan]

@Google Cloud Tech. When you guys show the demo on console, it doesn't benefit anyone. Most of the companies are doing it using IAS or using Python api. So please provide more tutorial using coding...

[havardestensen]

Too bad dead-lettering doesn't work for event-driven functions since you cannot manually NACK.