Lightning Talk: Migrating Prometheus data between different storage systems - Harkishen Singh

Lightning Talk: Migrating Prometheus data between different storage systems - Harkishen Singh, Timescale

"The Prometheus ecosystem includes many long-term data storage systems that allow users to store and analyze their Prometheus data. Each of these systems have different tradeoffs – and users sometimes want to switch between them as their needs evolve. But switching between these storage systems has not been possible because, until now, there has been no universal tool to migrate data between them.

Prom-migrator solves this problem. It is a universal, open-source Prometheus data migration tool that is community-driven and free to use. Prom-migrator migrates data from one remote-storage system to another remote-storage system, leveraging Prometheus’ remote storage API. This means that this tool can be used to migrate data to and from any storage system that supports the Prometheus remote_write/remote_read protocol (e.g. Promscale, Thanos, Cortex, M3DB, etc.).

Prom-migrator offers several key features:

Data migration from and to any storage system. This tool is designed to work with any remote storage system, so that users can migrate data across any system in a wide range of scenarios.

Informative outputs during runtime, allowing users to track progress. Prom-migrator keeps users informed about the migration progress, so that users can plan their time accordingly.

Ability to resume migration(s) in case of any unintended shutdowns. Keep a record of migration progress and, in the case failure or interruptions, automatically resume the migration where you left off.

Stateless working model. Easy deployment due to the fact that the migrator does not need to keep state. No need to worry about mounting volumes or attaching persistent storage.

In this talk, we will show how prom-migrator works, followed by a demo."