Exploring the Latency/Throughput & Cost Space for LLM Inference // Timothée Lacroix // CTO Mistral

// Abstract
Getting the right LLM inference stack means choosing the right model for your task, and running it on the right hardware, with proper inference code. This talk will go through popular inference stacks and set-ups, detailing what makes inference costly. We'll talk about the current generation of open-source models and how to make the best use of them, but we will also touch on features currently missing from the open-source serving stack as well as what the future generations of models will unlock.

// Bio
Timothée Lacroix, aged 31, is Chief Technical Officer in charge of technical issues relating to product efficacy and research. Started as an engineer at Facebook AI Research in 2015 in New York, where he completed his thesis between 2016 and 2019, in collaboration with École des Ponts, on tensor factorization for recommender systems. He continued his career at Meta until 2023 when he co-founded @Mistral-AI.

// Sign up for our Newsletter to never miss an event:

// Watch all the conference videos here:

// Read our blog:

// Join an in-person local meetup near you:

// MLOps Swag/Merch:

// Follow us on Twitter:

//Follow us on Linkedin:

[iandanforth]

There seems to be a mistake in the cost estimate at 21:53. It uses the price for the A10 but the throughput of the H100. I believe the actual cost estimate would be $48, not $15.

[mndflctzn]

This is awesome. Thanks for sharing super useful

[eduardoalvarez]

The math around 6:50 for A100 batch size isn't working out. It would be great if the values used to calculate the 400 batch size were provided.

Based on the equations provided for compute time and model load time, the point of intersection is Flops/(2*MemoryBand) NOT the (2*FLOPS)/MemoryBand which is in the video.

[frank]

Great talk! is there link to the slides for this talk?

[Gerald-izmv]

hi what benchmark he run to generate the plots? any open source github links?

[janilbolswong]

@5:40 why do we need to load the entire model all the time? can't we just load once? If so, we might lower the needs of memory movement, and the intersection would shift left

[boussouarsari]

It's possible that I'm misunderstanding, but given our use of a significantly large key-value cache (2GB multiplied by the batch size), can we still assert that the memory bandwidth is solely influenced by the model's weights?

[AbdulK-krjv]

What a horrible unethical response on the ethics of training data