Message Ordering in Pub/Sub or Queue

Do you need message ordering? Processing messages in order as they were sent/published to a queue or topic sounds simple but has some implications. What happens when processing a message fails? What happens to all subsequent messages? How is throughput handled when using the competing consumers pattern, which loses the guarantee processing in order. Lastly, is ordering even important?

🔗 Solace

💥 Join this channel to get access to source code & demos!

🔥 Don't have the JOIN button? Support me on Patreon!

0:00 Intro
0:41 Ordering Requirements
2:22 Competing Consumers
6:21 Failures
7:45 Processing Out-of-order

#softwarearchitecture #eventdrivenarchitecture #pubsub

[webbi]

Thanks. Now I kindof understand the partition when creating topic in kafka.

[raghuveerdendukuri]

I generally consider ordered processing for most of the systems, be it a payment system or a chat box or an order management system or a weather notification system etc

Limited usecases like "processing orders with abundant supply" accepts change in order


By defining boundaries in an optimistic way, different tasks can be sent to different channels/topics and even with single consumer per channel/topic will be helpful

[shahrazkl]

Thank you! Appreciate all the efforts you put into these videos!

[AhmedSamy]

Nice video as always, from what i have seen so far it's almost always avoidable problem, however, these are super ways to handle it when there's no way out :)

[pierre-antoineduchateau]

Very nice video as always Derek !
It seems to me that the saga is loaded the same way an aggregate is loaded in an event sourced system. Should a saga also have a key like the aggregate id ?

[emonymph]

Thanks for the video I liked and subscribed but feel like you rushed the most important and complex topic 'processing out of order', the illustration was quite poor and you lost me there. please spend more time or give two examples when going into a complex topic. I have no idea what a saga is and have never heard of it, would be nice to define it.

how to deal with failure gave me an idea, since kafka keeps a log would it be possible to code this: stop > consumer deletes messages received after failure > replay from the failed message using the log > continue as normal.

[vovaik]

Thanks.

There are some of message brokers which support message ordering feature:
* Message sessions in Azure Service Bus,
* Message group ID in Amazon SQS FIFO.

What else do you guys know?

[AndriiMarynych]

Thanks for the video, it brings an interesting perspective on how to avoid unnecessary additional technical complexity with in order processing!

But there's still one thing I wonder about. It's evident that something is wrong with ordering if you miss a message that's required for your business flow. But what about optional messages?
For example, if after an order is placed I change the delivery destination (DeliveryDestionationChanged event) and the flow doesn't process it, I might end up delivering to a wrong place. And considering that changing destination is an optional step, there's no easy way to notice that this step is skipped. Could you please share your experience on how to handle these situations?

[thedacian]

So assuring message order shall be message broker matter, not SAGA worker because in this case we shall be embed some business logic in the saga worker, right? Thank you!

[tharun]

Since network calls are involved during publishing how can we still maintain the ordering of events?

[zeamessaoud]

Great as always, Thanks Derek !
As far as I know rabbitmq does not natively support partition-key, do you have any trick?

[damilola_adegunwa]

hi Derek, what if the update events are sent to the consumer in the future just like the reserve-expire (delay-delivery) pattern you used in a previous video