Growing Pain: Scaling to 10K Workflows per Week - Yao Lin & Harris Khan, Bloomberg

Growing Pain: Scaling to 10K Workflows per Week - Yao Lin & Harris Khan, Bloomberg

At Bloomberg, we maintain a centralized multi-tenant workflow orchestration platform for internal engineering teams. This system has seen rapid growth in a short period of time, which has resulted in unique challenges as a workflow platform, including spiky submission rates, unpredictable resource requests at creation time. We also observe heavy demand on API server and workflow-controller. The necessary throttling and garbage collection adds further complexity to the situation. Moreover, our platform has a diverse user profile, as the company's engineers have varied cases and requirements. Fulfilling these requirements under the same cluster is another unique challenge, especially with a global Argo installation. This presentation will discuss some of these challenges and examines their potential impact. The speakers will share their observations and experiences that inspired some guardrails and mitigation ideas that are likely generalizable for other kinds of batch workloads.