filmov
tv
vLLM Office Hours - Distributed Inference with vLLM - January 23, 2025
Показать описание
In this session, we explored the motivation for distributed inference, delving into vLLM architecture and GPU parallelism to enhance performance. We discussed the challenges of serving large models, introduced the concept of tensor parallelism, and examined the benefits and trade-offs of leveraging multiple GPUs for inference. We also highlighted profiling tools for analyzing kernel performance and overhead, along with the potential challenges of adopting a disaggregated approach with separate nodes for prefill and decoding.
During the open discussion, we addressed various community questions, including practical applications of tensor parallelism in real-world scenarios, the impact of distributed inference on latency and throughput, and strategies for optimizing multi-GPU setups.
During the open discussion, we addressed various community questions, including practical applications of tensor parallelism in real-world scenarios, the impact of distributed inference on latency and throughput, and strategies for optimizing multi-GPU setups.
0:48:20
vLLM Office Hours - Distributed Inference with vLLM - January 23, 2025
1:19:57
[vLLM Office Hours #27] Intro to llm-d for Distributed LLM Inference
1:08:33
vLLM Office Hours - DeepSeek and vLLM - February 27, 2025
0:59:07
vLLM Office Hours #22 - Intro to vLLM V1 - March 27, 2025
0:04:58
What is vLLM? Efficient AI Inference for Large Language Models
0:35:53
Accelerating LLM Inference with vLLM
0:48:06
vLLM Office Hours - Disaggregated Prefill and KV Cache Storage in vLLM - November 14, 2024
1:04:28
vLLM Office Hours - Speculative Decoding in vLLM - October 3, 2024
0:05:42
Distributed LLM inferencing across virtual machines using vLLM and Ray
0:16:45
Run A Local LLM Across Multiple Computers! (vLLM Distributed Inference)
0:07:23
What is vLLM & How do I Serve Llama 3.1 With It?
0:56:04
vLLM Office Hours - vLLM’s 2024 Wrapped and 2025 Vision - December 19, 2024
0:00:48
Unlock LLM Speed: VLLM Crushes the Competition!
0:34:15
Inference with vLLM on Aurora
0:05:29
Real Demo: 𝐃𝐢𝐬𝐭𝐫𝐢𝐛𝐮𝐭𝐞𝐝 𝐯𝐋𝐋𝐌 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 (zero exposure) – By Super Protocol...
![[Ray Meetup] Ray](https://i.ytimg.com/vi/HDSy09hrm2I/hqdefault.jpg)
1:04:12
[Ray Meetup] Ray + vLLM in Action: Lessons from Pinterest and Large Scale Distributed Inference
0:30:52
The Evolution of Multi-GPU Inference in vLLM | Ray Summit 2024
0:56:09
vLLM Office Hours - FP8 Quantization Deep Dive - July 9, 2024
0:27:35
Distributed Inference with Multi-Machine & Multi-GPU Setup | Deploying Large Models via vLLM &am...
0:39:36
Scalable and Efficient LLM Serving With the VLLM Production Stack - Junchen Jiang & Yue Zhu
0:45:48
Optimizing LLM Inference with AWS Trainium, Ray, vLLM, and Anyscale
0:00:55
Next-gen distributed LLM inference 🌐 #ai
0:02:33
Unlocking vLLM The Future of Open Source Inference Servers
0:48:26
vLLM Office Hours - vLLM Project Update and Open Discussion - January 09, 2025
Комментарии