How We Scale a Distributed SQL Database to 1 PB - Jinpeng Zhang, PingCAP

How We Scale a Distributed SQL Database to 1 PB - Jinpeng Zhang, PingCAP

TiDB is a distributed SQL database that we built to solve the scalability problems of traditional SQL databases such as MySQL and PostgreSQL. Using TiDB, users do not need to shard their data across multiple MySQL or PostgreSQL database instances, nor do they need to sacrifice some key database features such as JOIN and transactions. Users only need to add storage nodes and computing nodes to the cluster as needed. However, we also encountered many scalability challenges when building TiKV - the stateful storage layer of TiDB. Challenges such as workload skew issues making it difficult to scale performance, management challenges of millions of dynamic data partitions, latency impact during scaling, interference between different workloads when consolidating multiple workloads into the same cluster, etc. In this talk, I will provide an in-depth look at these challenges and our solutions.