filmov
tv
Accelerated LLM Inference with Anyscale | Ray Summit 2024
Показать описание
At Ray Summit 2024, Anyscale Co-Founder and CTO Philipp Moritz, along with Cody Yu, present Anyscale's new LLM enterprise and production features, in addition to team's contributions to open-source inference engines.
In this talk, Moritz and Yu detail how the Anyscale team has collaborated with the vLLM open-source team, highlighting key advancements such as FP8 support, chunked prefill, multi-step decoding, and speculative decoding. They explain how these optimizations have led to significant performance improvements in vLLM, doubling both throughput and latency efficiency. The presentation also covers Anyscale-specific enhancements, including custom kernels, batch inference optimizations, and accelerated large model loading for autoscaling deployments.
This breakout session is a must-watch for anyone looking to gain insights into the latest techniques for improving LLM inference efficiency and scalability.
--
Interested in more?
--
🔗 Connect with us:
In this talk, Moritz and Yu detail how the Anyscale team has collaborated with the vLLM open-source team, highlighting key advancements such as FP8 support, chunked prefill, multi-step decoding, and speculative decoding. They explain how these optimizations have led to significant performance improvements in vLLM, doubling both throughput and latency efficiency. The presentation also covers Anyscale-specific enhancements, including custom kernels, batch inference optimizations, and accelerated large model loading for autoscaling deployments.
This breakout session is a must-watch for anyone looking to gain insights into the latest techniques for improving LLM inference efficiency and scalability.
--
Interested in more?
--
🔗 Connect with us:
0:29:35
0:45:48
0:27:39
0:35:53
0:30:28
0:31:35
0:03:18
0:27:30
0:38:11
0:32:07
0:30:52
0:32:27
0:37:03
0:30:19
0:30:25
0:35:23
2:03:28
0:32:12
0:27:59
0:31:02
0:06:15
0:32:21
0:30:51
2:05:06