E07 | Fast LLM Serving with vLLM and PagedAttention

Fast LLM Serving with vLLM and PagedAttention (SOSP'23)

Abstract: LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. To address this problem, we are developing vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention achieves up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes. vLLM has been developed at UC Berkeley and deployed for Chatbot Arena and Vicuna Demo for the past 5 months. In this talk, we will discuss the motivation, features, and implementation of vLLM in depth, and present our future plan.

Bio: Zhuohan Li is a CS PhD student at UC Berkeley, where he is advised by Professor Ion Stoica. He is interested in designing and building efficient machine-learning systems. Recently, he has been focusing on the training and serving of large models, specifically LLMs. His works include Alpa, AlpaServe, Vicuna, and vLLM (PagedAttention). He completed his BS at Peking University and has interned at Microsoft Research, Anyscale, and Google Brain.

[ginsongsong]

Thanks for the sharing. It’s educational for me.
One question, is the block size(16/32) related to the warp size(half-warp/warp)? Wondering the theory that you define the black size in kv cache.

[shabdanbatyrkulov]

Thanks for sharing!
Is it possible to turn on an automatic subtitles (with translation)?

[chenghao]

Any implementation that work with Azure?

[maciejgawinecki]

Is there a version with English speaking?