THIS is NVIDIA's New AI GPU for up to 10,000 GPU Installations ft. Supermicro

The NVIDIA L40S is the company's do-it-all GPU that is for professional visualization, vGPU capabilities, and even AI inference and training. We use Supermicro SYS-521GE-TNRT 8-GPU system to take a look at the NVIDIA L40S vs. the NVIDIA H100 and vs. the NVIDIA A100 to see why this GPU is a different option for model customization, generative AI/ LLM applications, as well as for video.

Thanks to Supermicro and NVIDIA for getting us the hardware to make this possible and flying me out to California for a few hours.

----------------------------------------------------------------------
Become a STH YT Member and Support Us
----------------------------------------------------------------------

----------------------------------------------------------------------
Where to Find STH
----------------------------------------------------------------------

----------------------------------------------------------------------
Where to Find The Unit We Purchased
Note we may earn a small commission if you use these links to purchase a product through them.
----------------------------------------------------------------------

----------------------------------------------------------------------
Timestamps
----------------------------------------------------------------------
00:00 Introduction
01:29 NVIDIA SXM5 H100 GPUs versus PCIe and OAM
04:27 Heritage of NVIDIA PCIe GPUs for AI
05:18 NVIDIA L40 vs L40S
06:22 NVIDIA L40S L40 A100 H100 H100 NVL PCIe Comparison Table
12:00 Wrap-up

[OTechnology]

As someone who got disappointed by switching to a dual 3090 setup from a dual Titan RTX setup, I just want to add that an important metric that Nvidia buries in the spec sheet that's important for DL performance is the FP16 with FP32 Accumulate. Where Nvidia half rates it on the Geforce cards to stop them from beating the pro cards.

[x]

As someone who works on these things, the only billionaire those GPUs will be making is Jensen Huang

[ProjectPhysX]

It's a bit too much toxic marketing for me. Can't build multi-GPU servers with cheap gaming cards anymore because Nvidia killed 2-slot 3090s/4090s to prevent exactly this. Their coolers are so heavy that they rip out the PCIE slot under their own weight, and so hilariously large they don't fit in even normal PC cases.
L40S is identical to the RTX 4090, except for the extra 24GB memory chips which cost $100 in retail, yet they increase the price by an extra $9000.
And NVLink on the A100/H100 is kept locked to their proprietary CUDA language, so you can't use it with portable OpenCL codes.

[Darkk]

Crazy amount of GPU computing power in one box.

[FlaxTheSeedOne]

By buying these GPUs I know someone who definetly is gonna be a Bilionare. But the more you buy the more you save.

[geofrancis]

This is where your 4090 shortage stems from....

[LerrySanders]

Thanks for the info Patrick. Great learning about things Ill be able to afford in 20 years when it hits ebay.

[trapexit]

Since I work in research the L40S just isn't a good choice. We need the VRAM. But for "smaller" or quantized LLM inference they can be fine.

Unfortunately, my data centers are not setup for high density servers so currently we are using 4x setups.

[xKruzr]

about to build a 40-node cluster with one of these in each machine. they're really remarkably flexible cards.

[Veptis]

For me, memory matters. H100 is 80GB, and RTX600 Ada, RTX5880 Ada, L40S and L40 are all 48GB. RTX4090 is just 24GB, but so is the RTX 3090ti, which is much cheaper.
There is water cooled RTX6000 Ada, but is there water cooled L40S?

Sadly, it's still not a good solution. Intel has nothing for workstation, AMD is missing software. At this point... I am hoping for Qualcomm to sell the Cloud AI100 Ultra with 128GB LPDDR5X for workstation

[JeffreyPeckham_abic]

Love the videos, love the content. Following data center tech has always been fascinating for me, even though it's been 10 years since I've had to step foot inside one.

I never thought I'd have to say this, considering I'm known for being an incredibly fast talker, but this was the first video I've ever had to slow down to 75% in order to not feel burned out.

[chromerims]

Unique coverage, fabulous 👍
Thank you STH

[Mpdarkguy]

Isn't vGPU/SR-IOV all software sauce? I really dislike nvidia price gating features that should be everywhere

[benjaminsmekens]

Just procured a bunch of L40S GPU's for our inference tasks, and they rip. We also have bunch of H100's for the initial training of our LLM's. You touched on it during your talk but power and cooling requirements are the biggest hurdle when it comes to the H/A100's. The L40S GPU's are a lot more reasonable, and the price and lead times are nice bonusses.

[AI-xijk]

Hi Patrick, thanks for the specs table. Makes it much easier to visualise. Nvidia publishes specs in a different format for different GPUs so it’s tedious to compare especially when naming is different. How does A800 stack up to these?

[blender_wiki]

If you are considering upgrading your models from an L40 to an L40S, it's important to observe that the tensor cores exhibit similar speeds on both cards at INT4 and you don't have any concrete adavantage. Additionally, if you currently use or own a few RTX6000 GPUs, upgrading to the L40S may not yield significant performance improvements; the differences in performance are marginal, except for certain specific aspects. However, if you are making a new investment, opting for the L40S could be a more favorable choice. It is advisable to carefully plan the amortization of this investment over an 18-month period.

[kwazar]

Looks like a nice workstation hedt to play games with.

[forutube]

I think I need the Torque Test Channel for these.

[geekinasuit]

Yeah the marketing was more than over the top in this one, but this is Nvidia for you. The L40S is a low-end device made to get around regulations limiting capabilities sold to China. Never-the-less, $300K for a few low-end GPU's is unreal, I can't see the price gouging continue for much longer now that competition is rolling in.

[chubbysumo]

wow, so they found a way to ship more consumer GPUs into the professional market, and have fully abandoned the consumer market. expect the 4090 prices to go wild, and probably completely go away. how long before they start doing it with the AD103 die too? fuck Nvidia for this.