Lightning Talk: OTel Me How to Build a Data Pipeline for Observability - Daniel Kim & Reese Lee

Lightning Talk: OTel Me How to Build a Data Pipeline for Observability - Daniel Kim & Reese Lee, New Relic

Observability boils down to a data problem. To gain observability, we need to collect signals from many different data sources, frameworks and programming languages. It is important to turn heterogeneous data across so many different signals into actionable insights through a framework like OpenTelemetry, an open source standard enabling us to collect, process, and export telemetry data. In this session, we will discuss the complexities that come with handling telemetry data from distributed systems as they scale. Next, we will show you how to build an observability data pipeline using the OpenTelemetry collector and open source plugins that can perform a wide variety of data processing, such as injecting infrastructure metadata into traces and metrics, implementing tail-based sampling, and exporting data to any backend via OTLP. Finally, we will share some of the challenges with running the collector, so that you can take them into account as you build out your own observability pipeline with OpenTelemetry.