Handling Failures in Message Driven Architecture

Many great libraries help to add resilience and fault tolerance by handling failures in a message driven architecture. However, it's not just as simple as adding retries, timeouts, circuit breakers, etc., globally to all network calls. Many implications are specific to the context of the request being processed. And in many cases, it's not solely a technical concern but rather it's a business concern.

💥 Join this channel to get access to source code & demos!

🔥 Don't have the JOIN button? Support me on Patreon!

0:00 Intro
1:09 Immediate Retry
2:05 Exponential Backoff
4:12 Deadletter
4:55 Circuit Breaker
6:47 Cascading Failures

#softwarearchitecture #messagedrivenarchitecture #eventdrivenarchitecture

[varshard]

Funny enough, I presented message driven architecture in my company's knowledge sharing session 2 days ago. Had you upload the video sooner, I could've just show your video and just eat popcorn.

[voduyquang]

Oh really I want to say thank you. You make the videos along with documents that very clear and more professional. The topics are very interesting nowadays. The big thanks for you pls keep it up!!!

[sathyajithps]

I was literally sitting down to create two queues for processing purchases in my on going project and this video dropped. What a timing! I always thought of just retrying or burying the message to the back of the queue or just creating a new queue for failed messages. Excellent video once again. 👌

[andersjuul]

Nice! This kicks off the day beautifully here in Denmark :)
Especially: Emphacising business involvement in how we should handle errors and the hold-up caused by retries within handling of a specific message.
Please, please keep it up! I love it! :)

[manan]

Glad i found your channel. Really advance concepts in such a nice and crisp video.

[cbaxtermusic]

you have the best, most consumable videos on very complex topics. I love this channel. I'm shocked you are not at 1mil followers yet.

[yamirtomas]

jesus man, you just keep making super interesting videos! keep it going! and thanks!

[glaydersen]

Really helpfull, as always! Thanks for the video!

[neha]

Hi where I can see practicle implementation..?

[tony-ma]

I can spin up multiple projections services (same projection but multiple instances in the kubenetes for mutiple processing) for one persistant subscription from the eventstore?

[edenr]

Very clear, I like the way you explain, keep it up 👍

[arikshapiro]

What would you do if you have a client that sent the initial command that started the flow asynchronously, but needs to know if a failure happened? would you have the client consuming from the deadletter queue to check if there's a matching message? or would you send a MessageFailed msg to the message broker yourself and have the client consuming those?

[carlosmauriciorebolledosie]

Hi, great content on this channel. I would like to know if there is a demo in code about how to build a dead letter queue mechanism, if I become a member could I find this demo?

[sviat]

Thank you for your videos!

I like the idea of loosely coupled monolith but I think in real world too few projects starts with this approach. Could you share your thought about strategies how to move out from coupled monolith when it is live on production? Have you had such cases in your practice?

[bruno.arruda]

Nice video!
Do you have some vision or any draft of how potentially Distributed Circuit Breaker could be?
Another service between the multiple instances and external services? Or any framework/library already doing such a thing?
Thanks!

[allinvanguard]

Great video! I have a question - What is the usual process of continuing with deadlettered messages? Are these messages scheduled by the queue after X amount of time? Would the consumer actively try to re-query these messages once it knows that is is now capable of processing them?

[varshard]

A trick for an exponential back off that doesn't hang up your worker node that I did was using multiple queues as a sort of psuedo dead letter queues.

1. Create a few queues and set the working queue as their dead-letter queue.
2. Set different message TTL based on each back-off period on each queue. So, when a message is expired, they will be requeue in the workin queue.
3. When a fail message on a consumer node, the node will increment nuimber of retries in the message header, then throw the message to the psuedo dead letter queue. ie: retries = 1, send it to the queue 1.
4. After the message is published to a psuedo dead letter queue, the consumer will ACK message back to the working queue toT remove the message.

You need multiple queues with different TTL instead of setting TTL on each message, because TTL on each message will be taken into account only when the message is at the head of a queue. So, a message with short TTL have to wait for a prior message with a longer TTL to be expired first. A head of the line issue.

[vinylwarmth]

Appreciate this video, thanks. Doing a lot more messaging in my latest role so finding your videos extremely useful 👍

I'm notice you use Kafka in many of your videos. Have you got a blog post/video on why or how you'd choose between Kafka/Service bus/Event grid? I'd be interested to learn about that