Conditional Distributed Tracing

Presented at o11ycon+hnycon, June 9–10, 2021

Will Sargent, Software Engineer @ eero

Conditional Distributed Tracing

Distributed tracing is generally a binary affair—it's off or on. Either a trace is sampled or, according to a flag, it's not. Span placement is also assumed to be an "always-on" system where spans are always added if the trace is active. For general availability and service-level objectives, this is usually good enough. But when we encounter problems, we need more. In this talk, I'll show you how to "turn up the dial" with detailed diagnostic spans and span events that are inserted using dynamic conditions. The talk involves Java/Scala code examples, use of feature flag systems, and examines several possible dynamic conditions like time (for the next 10 minutes), error (the system's unhappy), or target condition (user's session, http user agent, etc).