Fabian Reinartz: The new Prometheus storage engine

"From the beginning, Prometheus was built as a monitoring system with Cloud Native environments in mind.

Orchestration systems such as Kubernetes are rapidly gaining traction and unlock features of highly dynamic environments, such as frequent rolling updates and auto-scaling, for everyone. This inevitably puts new strains on Prometheus as well.

In this talk we explore what the challenges are and how we are addressing them by building a new storage layer from the ground up.

The new design enables consistent performance with thousands of concurrent writers inserting millions of new samples per second on a single machine.

We will discuss how the new approach significantly reduces resource requirements for CPU, memory, and disk IO alike. At the same time its more efficient indexing techniques increase query performance and allow us to gracefully handle high turnover rates of monitored application instances."

Bio: Fabian is a software engineer at CoreOS, where he works on the Prometheus open source eco-system and its integration with Kubernetes. Previously, he was an engineer at SoundCloud.