AI/ML/DL GPU Buying Guide 2024: Get the Most AI Power for Your Budget

Welcome to the ultimate AI/ML/DL GPU Buying Guide for 2024!

In this comprehensive guide, I'll help you make informed choices when selecting the ideal graphics processing unit for your AI, Machine Learning, and Deep Learning projects.

🚀 Whether you're a data scientist, researcher, or just an enthusiast, I've meticulously evaluated nearly 70 GPUs and summarized their AI capabilities. My aim is to ensure that you get the most AI power for your budget in 2024.

🔍 What to Expect in this Video:

- Guidance on how to choose a GPU for AI/ML/DL
- Price-performance comparisons to empower your decision-making
- Expert recommendations tailored to different AI/ML/DL requirements

By the end of this guide, you'll have a clear understanding of which GPU is the perfect fit for your AI-related tasks in 2024, enabling you to maximize your investment's value. If you're passionate about AI and want to make the most of your budget, this video is a must-watch. Don't forget to like, subscribe, and hit the notification bell to stay updated on our upcoming AI/ML/DL content!

📚 Additional Resources:

Guide For Choosing AI/ML/DL GPU
GPU Comparison - Excel Data
Interactive GPU Comparison - PowerBI Report
AI/ML/DL with the Dell PowerEdge R720 Server - Energy, Heat, and Noise Considerations
Throttle No More: My Strategy for GPU Cooling in Dell PowerEdge
Installing Tesla P100 GPU on Dell PowerEdge R720 Server with Driver Installation
Installing DUAL Tesla P100 GPU on Dell PowerEdge R720 Server with Driver Installation
Dell PowerEdge R720XD GPU Upgrade: Installing Tesla P40 with NVIDIA Drivers
Dell PowerEdge R720 GPU Deep Learning Upgrade: Installing Dual Tesla P40s with NVIDIA Drivers

HOW TO GET IN CONTACT WITH ME

🐦 X (Formerly Twitter): @TheDataDaddi

Feel free to connect with me on X (Formerly Twitter) or shoot me an email for any inquiries, questions, collaborations, or just to say hello! 👋

HOW TO SUPPORT MY CHANNEL

If you found this content useful, please consider buying me a coffee at the link below. This goes a long way in helping me through grad school and allows me to continue making the best content possible.

Buy Me a Coffee

As a cryptocurrency enthusiast, I warmly welcome donations in crypto. If you're inclined to support my work this way, please feel free to use the following addresses:

Bitcoin (BTC) Address: bc1q3hh904l4uttmge6p58kjhrw4v9clnc6ec0jns7
Ethereum (ETH) Address: 0x733471ED0A46a317A10bf5ea71b399151A4bd6BE

Should you prefer to donate in a cryptocurrency other than Bitcoin or Ethereum, please don't hesitate to reach out, and I'll provide you with the appropriate wallet address.

Thanks for your support!

[starlordhero]

00:11 Choose GPU with newer architecture for better performance
02:28 Choose NVIDIA GPUs with active support and sufficient VRAM for future scalability.
06:51 Key considerations for choosing an NVIDIA GPU for deep learning
09:08 Consider driver support for deep learning framework compatibility.
13:10 Factors to consider when choosing an NVIDIA GPU for deep learning
15:12 Understanding the key GPU metrics is crucial for making the right choice.
19:46 Choosing GPU based on performance, memory, and bandwidth criteria.
22:00 GeForce RTX 2060 Super and GeForce RTX 4060 TI 8 Gbit are the best bang for your buck GPUs.
26:27 Comparison of NVIDIA GPU models for Deep Learning in 2023
28:45 GeForce RTX 4060 Ti 16GB has the best raw performance
33:18 Choosing NVIDIA GPUs for Deep Learning in 2023
35:36 Best bang for your buck: P100 and P40 GPUs
39:22 P100 and P40 are recommended for deep learning
Crafted by Merlin AI.

[jeffm]

Brother - you went down a serious rabbit hole. A man after my own heart. One thing you didn't mention that is very important to data integrity and therefore results is ECC RAM. The pro Nvidia GPUs have ECC RAM. Most casual users don't realize how many bits are flipped through flaws in silicon and cosmic events (literally). Then some garbage gets written to disk. That's why I would never own a workstation that doesn't have ECC from top to bottom. Better if the OS / Hypervisor is using ZFS (the best file system IMO... with 40 years building enterprise and global systems). Consumer equipment is fine to test and learn on, especially on a budget. But if you want data and result integrity, at a minimum, buy pro equipment.

Like you - I've had great results buying refurbs from eBay. Used electronic prices drop faster than pulling out of an auto sales lot in a new car. Well - at least that used to be true. But it still is for workstations and servers. I recently looked at some Dell PowerEdge 730's (NVMe M2 bootable) with 128 GB ECC RAM and dual, upper-end, v3 Xeon processors for about $400 - with IDRAC (out of band management).

You did emphasize the use case that the GPU is for, and that's what I'd emphasize too. If the data you are processing isn't something that you can't replace with numerous backups, or can't suffer glitches - go with professional equipment - either new or used.

Then use ZFS RAID and not hardware RAID. ZFS controls the whole data stack - from RAM to permanent storage. You'll want to disable hardware RAID so firmware doesn't fight with ZFS. If you are learning and experimenting, but not relying on the end result - use cheaper consumer products. The learning curve is lower and so is the price tag. If you've never managed a PowerEdge server - that is an entirely different animal because it has to be. The difference between one of those and a consumer PC is like the difference between a flip phone and a Linux workstation. Night and day. But if you're a nerd like me - that's what you want as a platform.

[RSV]

What most people don't think about is that time is money !

[pliniosborges]

I wish I had watched this video 1 day ago. Great material for beginners in ML. Thank you. I ended up choosing a RTX 4070 12gb. Not the best choice for money, but I guess still very powerful

[prentrodgers]

This is a fantastic video explaining how to choose a GPU for deep learning/AI/ML. He extended Tim Dettmers single GPU performance chart into a masterpiece of a spreadsheet and PowerGI dashboard. Masterful. I wonder if you factored electricity cost, the cost of removing heat from the room, and total decibel output into the decision. I see in a subsequent video that the server is installed in what looks like a basement. The rack is within a few feet of a gas can. Those Dell machines can run hot, so you might want to move the gas can elsewhere. How noisy is the final product with two P100's?

[bdhaliwal]

Your insight and the spreadsheet you provided are invaluable. Thanks.

[whomair]

Just what I was looking for! Thank you for all the hard work in putting this together - you make the world a better place :)

[kidrock]

I'm watching this as a newbie from a hotel room on my laptop with sub-par speakers. Just 1 request from my unique context would be to amp up the volume on future uploads so it is easier to listen when in similar situations.

[benbencom]

Nice work. I think the only big flaw I see in your analysis is that purchase price is not the entire upfront cost. Each card should have an overhead cost based on the fraction of a chassis, mobo+cpu, and PSU it would use. I think you just had a chassis for 2 cards as a sunk cost in your mind so it didn't matter. But for anyone building a full system (or systems) it would have a big impact on their purchasing decision.

[DoomhauerBTC]

Excellent video. I very much appreciate the time and research that you put into this.

[gax]

Thank you brother for your hard work. You have saved me a lot of time. Your spreadsheet is amazing! We can sort out GPUs by the desired category! I believe that, after viewing your results, the GeForce GPUs that are most notable (especially considering the price) is the 3080 Ti. It is close in CUDA cores to the 3090 (I am aiming for a 3090, but it might be easier and cheaper for me to get a second-hand 3080 Ti)

[nahuelgarciaoronel]

Thank you SO much, you've saved me hours of life down this rabbit hole!!

[melaronvalkorith]

Seriously great video man. I can tell this was a labor of love. Thank you for taking the time to create this!

[Marioz]

The work you did to put this together is very much appreciated. Thank you for the thorough and thoughtful analysis!

[miguelpineda]

Great video, I'm very grateful; it was worth watching in its entirety. Thank you for your effort, greetings from Panama. Your video will greatly assist me in a project that my classmates and I want to undertake at the university. Many thanks for sharing such valuable information

[aidanm]

I live for this deep dive stuff. Thanks for your thorough work!

[rukitorin]

Im torn in choosing for Gpu for ai use first (koboldccp + sillytavern) and gaming second. My choices were a 3060 12gb at first then the 4060 ti 16 stood out but then the 4070 ti super got recommend to me. I intend to use the card for at least 3-5 years. The only thing limiting me is my small budget. Like i could buy the 3060 now and the 4060 ti after few weeks. While ill wait and watch out for deals on the 4070...

[samjco]

It would be nice to add in energy consumption, heat vs cool, most durable, and real life comparison using an actually local LLM system (and which LLM size) with these cards. For instance, the best card for single use performance, vs multi use performance in using a local LLM system like privateGPT, for an example.

[deineMudda]

I like to throw in a perspective, currently im using an GTX 960 2gb vram, 7 gb ddr 2 ram @ 600mhz, I also have shared video ram or what it was called which ups my "vram" up to around 5gb (can be checked in task manager for example), AMD tripple core @ 2.1ghz, im fine tuning the smallest gpt2 model via CUDA at around 1 iteration per ~30 seconds and it uses around 3.5gb of my "vram".

[noth]

GFLOPS is not calculated like shown in the video at 15:18, remove the Giga which we know, FLOating Point operations per Second, simply(there is some history for why this is used). It is somewhat archaic since a lot if other things are being done too, which aren't incorporated in this but in general most other operations take less cycles than a floating point one does because the comma needs special attention so to speak, 15*15 and 1, 5*1, 5 are the same thing except for tracking the comma separately with the result being 225 or 2, 25. What I mean is the circuit needs additional logic to track commas or rather fractions so to speak, which is why we separate floating point from integer operations - additional hardware is required to track the comma "in top of" the integer type numerical operations. No idea if this makes any sense or is useful, I thought it would be simple to explain until I thought it through and realized I need to type this as opposed to scribble and show on a white board. I'm sure there is a good explanation for it out there, just trying to point to why since to a person doing math it's not as obvious as it is designing a circuit to do it.