Centralized vs Decentralized Prometheus Scraping Architecture... - Rabun Kosar & Ales Koprivnikar

Centralized vs Decentralized Prometheus Scraping Architecture with DoorDash - Rabun Kosar, Doordash & Ales Koprivnikar, Chronosphere

There are two primary approaches to scrape and collect metrics using Prometheus - using a centralized set of dedicated scrapers or decentralized scrapers that run as an agent. With centralized scraping, Prometheus is deployed as a central scraper to pull metrics from all discoverable endpoints and sometimes can be split across multiple centralized instances using a few different approaches. However, with a decentralized approach, Prometheus runs as an agent and in Kubernetes is deployed as a DaemonSet on each node in a cluster and only collects metrics from the node it runs on. Each model has pros and cons - especially when operating at large scale - which can make it difficult when deciding which model to use. In this session, Rabun and Ales will provide an overview of Prometheus metrics collection at DoorDash, where having highly reliable resources, easy endpoint discovery, and real-time insights is critical. They will share insights and best practices into DoorDash’s decision to implement a decentralized model by offering pros and cons of each approach. Leave with a better understanding of the “right” model for your use case(s).