How to use OpenTelemetry to Trace and Monitor Apache Kafka Systems

Tracing each request and its journey to completion in Kafka gives companies access to invaluable data that provides insight into system performance and reliability. Furthermore, using this data allows engineers to quickly identify errors or anticipate potential issues before they become significant problems. With greater visibility comes better control over application health - all made possible by OpenTelemetry's unified APIs and services.

By leveraging OpenTelemetry, organizations can ensure their applications and systems are secure and perform optimally. It will quickly become an essential tool for large-scale organizations that need to efficiently process massive amounts of real-time data. With its ability to scale independently, robust analytics capabilities, and powerful monitoring tools, OpenTelemetry is set to become the go-to platform for stream processing in the future.

Roman explains that the OpenTelemetry APIs for Kafka are still in development and unavailable for open source. The code is complete and tested but has never run in production. But if you want to learn more about the nuts and bolts, he invites you to connect with him on the Confluent Community Slack channel. You can also check out Monitoring Kafka without instrumentation with eBPF - Antón Rodríguez to learn more about a similar approach for domain monitoring.

EPISODE LINKS

TIMESTAMPS
0:00 - Inro
4:14 - What is OpenTelemetry?
7:52 - Tracing vs. Logs
11:26 - Three ways to do application-level tracing with OpenTelemetry
15:47 - What can you do if OpenTelemetry's agent doesn't support a specific API?
17:57 - What's missing in OpenTelemetry's native Kafka support?
32:29 - What can you see when using OpenTelemetry?
36:10 - Getting started with OpenTelemetry for event-level tracing
39:14 - Synchronous vs. Asynchronous processes
48:13 - It's a wrap!

ABOUT CONFLUENT

#streamprocessing #apachekafka #kafka #confluent

[brijeshjaggi]

wish it had a demo, with this talk..

[francksgenlecroyant]

I loved everything, the background music, the content and how the Kafka ecosystem is growing bigger!

[javadsaljooghi]

wish to have this tracing for kafka Stream and connect sooner, in out company we use them and we need to trace messages and monitor for latency and throughput based on message, Can you also make a podcast with demo how is instrumentation with prometheus exporter or JMX or JFR for kafka stream and connect? there is not enough information on how to do business logic(App) instrumentation in kafka and which has less overhead and preffered?!

[bjego-the-dev]

Well - I would have thought that the confluent cloud could on the serverside sends the metrics and logs directly to an opentelemetry collector like jaeger or application insights. It's pretty poor that they only support prometheus and proprietary platforms like splunk and datadog..
With this approach each development team needs to integrate opentelemetry for their client on their own.