Redesigning Scheduling in Ray to Improve Cost-Efficiency at Scale

Scheduling is a key component of making AI applications cost efficient at scale. This talk discusses the key challenges of building a scheduling system for AI applications: diversity of application requirements and performance. We will walk through two Ray AI applications: model serving and data preprocessing for distributed training and show how Ray scheduling makes them run faster and cheaper. In this talk, we will cover several Ray scheduling features including placement groups, graceful node draining and label based scheduling.

Takeaways

• Audiences will understand different ray scheduling features

• How to use them for their requirements

• How to make their applications run faster and cheaper.

About Anyscale
---
Anyscale is the AI Application Platform for developing, running, and scaling AI.

If you're interested in a managed Ray service, check out:

About Ray
---
Ray is the most popular open source framework for scaling and productionizing AI workloads. From Generative AI and LLMs to computer vision, Ray powers the world’s most ambitious AI workloads.

#llm #machinelearning #ray #deeplearning #distributedsystems #python #genai