What runs ChatGPT? Inside Microsoft's AI supercomputer in 2024 | Featuring Mark Russinovich

Microsoft has built the world’s largest cloud-based AI supercomputer that is already exponentially bigger than it was just 6 months ago, paving the way for a future with agentic systems.

For example, its AI infrastructure is capable of training and inferencing the most sophisticated large language models like GPT-4o at massive scale on Azure. In parallel, Microsoft is also developing some of the most compact small language models with Phi-3, capable of running offline on your mobile phone.

Watch Azure CTO and Microsoft Technical Fellow Mark Russinovich demonstrate this hands-on and go into the mechanics of how Microsoft is able to optimize and deliver performance with its AI infrastructure to run AI workloads of any size efficiently on a global scale.

This includes a look at: how it designs its AI systems to take a modular and scalable approach to running a diverse set of hardware including the latest GPUs from industry leaders as well as Microsoft’s own silicon innovations; the work to develop a common interoperability layer for GPUs and AI accelerators, and its work to develop its own state-of-the-art AI-optimized hardware and software architecture to run its own commercial services like Microsoft Copilot and more.

► QUICK LINKS:
00:00 - AI Supercomputer
01:51 - Azure optimized for inference
02:41 - Small Language Models (SLMs)
03:31 - Phi-3 family of SLMs
05:03 - How to choose between SLM & LLM
06:04 - Large Language Models (LLMs)
07:47 - Our work with Maia
08:52 - Liquid cooled system for AI workloads
09:48 - Sustainability commitments
10:15 - Move between GPUs without rewriting code or building custom kernels.
11:22 - Run the same underlying models and code on Maia silicon
12:30 - Swap LLMs or specialized models with others.
13:38 - Fine-tune an LLM
14:15 - Wrap up

► Unfamiliar with Microsoft Mechanics?
As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.

► Keep getting this insider knowledge, join us on social:

GPT-4o is the large language model used behind Apple Intelligence and updates to Siri.

#AI #AISupercomputer #LLM #GPT

[alexpearson]

This is my favorite video that Microsoft makes. So cool

[blitzio]

Awesome to see this, especially the hardware, networking and data center breakdown and info.

[ds]

That’s why I choose to buy their stocks, they know what it means to actually work. It was a long way for me from early 90s, when I’m - hardcore Unix user was calling Windows only using words “must die”, to start spending my free money on their stocks, and to actually admit what this company is really doing all this time. Thank you guys for keeping that spirit!

[Breaking_Bold]

Very very informative…sent it to my kid who is in college to see and keep seeing till they understand every word!!!

[Daniel-esdq]

I’m so glad people much smarter than I are working on this.

[bz]

With Great Power comes Great Capabilities...

Microsoft 📲💻🖥🎮

[ABLwAmazing]

Ah, the sysinternals guy. I owe half my career to this guy. Thx.

[liberty-matrix]

"it's funny you know all these AI 'weights'. they're just basically numbers in a comma separated value file and that's our digital God, a CSV file." ~Elon Musk. 12/2023

[GhostyDog]

What’s that again..? You’re adding the capacity of the third most powerful supercomputer every month! 😮

[drivenbycuriosity]

Most fascinating part for me is the Multi-LORA.

[ShpanMan]

Underrated video, a lot of cool useful details!

[Rafael]

So they can now run the same LLm on different GPUs(Nvidia vs Maya vs AMD)?

[IshaqIbrahim]

Timeline: 9:00 What happen to the heat energy extracted during cooling? Does it get used to generate electricity to power other devices or supply energy to some of the cooling fans or is it not used for anything?

[LouSpironello]

Great info about the architecture! Thank you.

[duran]

Did I understand correctly: "Today, 6 months later, we deploy the equivalent of 5 of those supercomputers every month"!?!?

[lifeslooker]

What would it take to take a 175B model to shrink it to run on a mobile phone? What are the limitations? The language used in the model? Can a compression be used or a language be developed that doesn't take up much space?

[QuantumXdeveloper]

Great session, Mark is as always the best❤

[Jj-duls]

5 times the Azure supercomputer deployed each month? Is that a typo..

[sachoslks]

5 times the Azure supercomputer deployed each month, thats insane!!! What does that mean for training next gen frontier models? 30x November 2023 does it mean you can train it 30x longer, 30x bigger or 30x faster or what? Will this continue up to the end of the year reaching almost 65x compute in one year?

[jamieknight]

It’s amazing… impressive budget for by chips from NVIDIA. But is it worth it? Curious to see if AI will take off or not.