The CORRECT way to implement Retries in .NET

Hello everybody I'm Nick and in this video I will show you how you can make your code more resilient by implementing retry policies. Retry policies power pretty much every microservice out there in some capacity, but due to the nature of having multiple systems coming together and needing to deal with things that might fail.

Don't forget to comment, like and subscribe :)

Social Media:

#csharp #dotnet

[robertmckee]

I've been using Polly for a long time. One of the things you have to be careful of though is when making HttpRequests and getting a timeout. If your request should mutate data, and the request is not idempotent, then recalling the method on a timeout (that actually succeeded) during a retry could cause duplicate calls. Most of my API work unfortunately falls under that umbrella, so I don't set a global retry policy, and handle each call individually.

[davestorm]

Very good to know this info. I do approach things differently: I have a background service that polls external APIs at regular interval and caches the data locally. Then, my API will send cached data to my clients. My polling service would do the retries in the background and if, for some reason, fails, the client's experience is not affected as they get the last cached data. This is great for a couple of reasons:
Client experience is not interrupted nor delayed.
External calls are only made at predefined intervals - so light on the provider's end (no "punishments" for hitting their service too hard).
External API usage costs based on hits/traffic can be reduced greatly, thus saving money: Say you had a 3rd party service that allowed 5000 hits per month for $100, and the next level was 50K hits for $500, and, without pre-polling/caching, you were calling ~30K hits per month. You would have your background poll every 10 minutes, or 4320hits/mo and save $400 every month - right off the bat. It would also save on bandwidth (while making it far more predictable), and with cloud services nickel-and-diming you to death on just about everything, you might save a few hundred more dollars every month on top of that! I love cloud tech, but it's still absurdly expensive and there's always those invoice shockers. Caching data is a good way to mitigate some cloud-sticker-shock.

[jessica_tee]

Love this channel. just started a new role where I am using Polly, had a rough idea of what was going on but not a full understanding, totally get it now, including Jitter which I seen being used but didn't really know what was going on! Thanks :)

[HomeSlize]

Polly is one of my favorite packages. I can't image life with out it.

[DamianWalczak]

I came here to see polly implementation and that is excactly whats there, kudos

[jonasgranlund]

Great content as usual, I remember we talked about the jitter a couple of years ago when you did a video on retries. Regarding status codes to retry on, I think 429 that for instance APIM deliver if you add the Throttling Policy can be good to retry on. Also, when I get the 429 from Cosmos I first retry a couple of time with the RetryAfterMs that the DocumentException property deliver and if I dont succeed I respond to caller with a Responseheader that have the RetryAfterMs value so the Client can retry after a "recommended" wait time. Same with 403 Forbidden, some APIs incuding google use 403 instead of 429 if a client call them to many times in a short period so that can also be good to retry on against some APIs, while against other APIs it can be a waste of resources... Happy Coding! :)

[jakesurrett]

Love this style of production-ready solution to common problems that are easy to mess up on your own. Same goes for the working with secrets video. Considering how many current and future software disasters you've enabled your viewers to avert. That's more than just good content - it's good karma.

[andreybiryulin]

Really proper way for retries is to implement circuit breaker pattern. Jitter and exponential retries just smooth out load on downstream API, but it almost doesn't reduce number of requests which you will fire into it when it fails.

[alexp]

Please don't use goto in public. A lot of guys, especially newbies don't even know it exists and this is so good

[dolaudz]

I like that the 0.69 and 420 were not left out. Nice.

[iSoldat]

I've used Polly in the past; simple and elegant.

[inth]

Your content is always top notch! Can´t wait to watch your take on the Circuit Breaker. It was the first ¨resilency pattern¨ I learnt and I think it´s one of the hidden gems when talking about resilency.

[andrewshirley]

I'll say that this scenario probably isn't appropriate for a retry anyway. If your API simply proxies another API, and the other API is down, then in reality your service is down too. Delegate the retry logic to the client, because the client can determine what kind of UX it wants. Is it okay with a user waiting 16 seconds as it spins? Is it going to throw up an error message on immediate failure and passively retry in the background? Is it going to stop trying and prompt manual reattempts at some point? They're all fine, but the thing is the backend doesn't need to worry about it at all, it does its one-time proxy and if it fails, it fails. A circuit breaker of some sort may still be appropriate to avoid contributing to an accidental DDoS, because a client doesn't have to respect your retry policy anyway and can just spam requests until you DDoS anyway.

There are situations where this kind of resiliency *is* more important, though, and the concepts you demonstrated here + the use of polly is great. Maybe in aggregation requests where your API is calling 4 or 5 APIs and you don't want a transient failure in one of them to kill the entire request is probably the best example. Another popular one is if you expect call chains across dozens of services waiting for your final return, but if that's what the company has built then then murder is probably a more appropriate solution than retry policies. Passive background processing services with no user on the other end are probably acceptable as well, but such a process is probably already on an infinite loop of some sort anyway so it really doesn't matter there.

[andreisalagean]

Any plans to add a new course anytime soon ? Maybe something about microservices ? They are so popular nowadays and hard to find some good explanation about properly implementing them covering failure cases and scaling.

[stratosmav]

Nice video... as always. I was also wondering whether you are planning to make a video about unsafe performance optimizations that can be done. Even though they might not be used regularly, they would still be very interesting.

[dakotapearl]

I'm really looking forward to see what other videos you come up with on the subject of resiliency. It's so often just retries, but so extra topics would be amazing!

[GregUzelac]

This is really helpful. I love your enthusiasm and knowledge. Thx.

[nove]

Great insight, i was on the fence about using polly

[__alexfox__]

Thank you! This is exactly what I need in my project right now!

[pquesadacr]

I really loved this video... Thank you so much Nick!