Debug your distributed service easier. Based on experience as an Amazon Principal Engineer

Over the last twenty years most of my work has shifted to distributed systems, especially web services. My first large-scale service debugging at Amazon was harrowing! I tell a bit about what makes it hard, and how a little foresight in your own system design and implementation will make your life MUCH easier during on-call / dev-ops.

TABLE OF CONTENTS
00:00 Introduction
01:07 DNS and BGP
04:07 Amazon, SOA, and me
05:47 Amazon service debugging story
10:23 a GUID helps
12:31 Best practices
13:47 Conclusion

[ai_university]

I worked for RedHat as a Senior OpenShift (Kubernetes) Consultant. Some of my customers were places like Ally Financial, Discover Financial, and Johnson Controls. The problems you described were ones I encountered often. Attaching a GUID to every call to every service in the chain is in fact the most often sought solution. Interfaces like Jaeger, Hubble, and OpenTelemetry/OpenSearch make it very snappy to trace the flow of traffic for each call.

[Itachi.Uchiha.Offical]

Sad that an Amazon Principal Engineer gets less views than those eX-fACeBoOk EnGinEeRs with 3 months of experience. Keep up the good work!

[faence]

We are designing a new distributed service, I am definitely implementing this GUID pattern. Thanks for saving our team from a future of headaches!

[Middleclassbeta]

This video was really informative. Can you also make a video on your experience on the best practices while breaking up your services. There are so many ways to think about it, but there must be some checklist that one can refer to.

[parthmaheshwari]

Who should be responsible to create GUID?

[mohabkhaled]

And we are up, the technicals started hard as it should 😂,
Thanks John for the story,

[shantanushekharsjunerft]

Great content but don’t see anything new lately!

[theyruinedyoutubeagain]

Corporate engineer discovers tracing lol