Databricks' vLLM Optimization for Cost-Effective LLM Inference | Ray Summit 2024

preview_player
Показать описание
At Ray Summit 2024, Megha Agarwal from Databricks (MosaicML) presents their team's work on enhancing vLLM for improved LLM inference performance. The talk focuses on Databricks' efforts to achieve industry-leading cost and performance in their LLM serving product.

Agarwal delves into the challenges of optimizing vLLM, particularly addressing GPU blocking operations during decoding steps. These operations, including sampling, input tensor preparation, and output processing, can account for about 30% of each step at large batch sizes for 70B+ models.

The presentation covers Databricks' solutions to reduce GPU idle time and accelerate quantization using custom kernels. Agarwal shares insights from their experience as vLLM developers, discussing future optimization areas and offering best practices for benchmarking.

This session provides valuable information for organizations and developers working on large-scale LLM deployment, offering practical strategies to enhance inference efficiency and reduce costs.

--

Interested in more?

--

🔗 Connect with us:
  1. Anyscale
Рекомендации по теме
Databricks' vLLM Optimization 0:27:39

Databricks' vLLM Optimization for Cost-Effective LLM Inference | Ray Summit 2024

Accelerating LLM Inference 0:35:53

Accelerating LLM Inference with vLLM

Model Alignment at 0:32:34

Model Alignment at Scale using RL from AI Feedback on Databricks

Scaling Video Ad 0:23:35

Scaling Video Ad Classification Across Millions of Classes with GenAI

LLM2 Module 3 0:04:21

LLM2 Module 3 - Deployment and Hardware | 3.5 Multi-LLM Inferencing

What is Retrieval-Augmented 0:06:36

What is Retrieval-Augmented Generation (RAG)?

How Large Language 0:05:34

How Large Language Models Work

vLLM Office Hours 0:52:35

vLLM Office Hours - Advanced Techniques for Maximizing vLLM Performance - September 19, 2024

LLM in Practice: 0:40:32

LLM in Practice: How to Productionize Your LLMs

A Practitioner's Perspective 0:36:47

A Practitioner's Perspective on LLMOps at DoorDash

vLLM Office Hours 1:13:14

vLLM Office Hours - Using NVIDIA CUTLASS for High-Performance Inference - September 05, 2024

Scaling RAG and 0:29:05

Scaling RAG and Embedding Computations with Ray and Pinecone

The State of 0:35:23

The State of vLLM | Ray Summit 2024

Vector databases are 0:03:22

Vector databases are so hot right now. WTF are they?

LLM Module 0 0:14:22

LLM Module 0 - Introduction | 0.2 Why LLMs

Faster and Cheaper 0:28:04

Faster and Cheaper Offline Batch Inference with Ray

How to Build 0:19:06

How to Build a Multi Agent AI System

Accelerated LLM Inference 0:29:35

Accelerated LLM Inference with Anyscale | Ray Summit 2024

Sponsored by: Accenture 0:35:02

Sponsored by: Accenture | Building GenAI-Powered Solutions using Mosaic ML, DBRX LLM and LLMOps

Advancing AI 2024 2:11:09

Advancing AI 2024 Replay

Building an LLMOps 0:43:28

Building an LLMOps Stack for Large Language Models | LLMs

FrugalGPT: Better Quality 1:02:59

FrugalGPT: Better Quality and Lower Cost for LLM Applications // Lingjiao Chen // MLOps Podcast #172

Orby AI + 0:48:49

Orby AI + Databricks Discuss How Gen AI Will Define the Future of Work

An AI stack: 0:58:25

An AI stack: from cloud orchestration to LLM evaluation