How Fast Will Your New Mac Run LLMs?

How fast can the new Apple Silicon Mac you so desperately want run LLM's and is it worth the price?

00;00 Intro
00:47 Benchmarks
05:06 Unbox
05:47 Results

Support My Work:

[ernestoflores]

Wow nice video! Very helpful, and well explained. Thanks

[blakenoh]

Great vid. Would you say there's a need for 96 GB RAM on a m2 or would 64 GB be enough for big models like mixtral

[gideonspace]

Hi Ian, um.. this year i will be going to college for my btech degree for 4 years and its on artificial intelligence and data science . In this course their will be core subjects of computer science as well as on ai and ds. I will be dealing with machine learning, computer vision, deep learning, etc and I plan on doing competitive programming in my second year. As well as I would love to have additional functionalities like using davinci resolve and blender, etc but that's just well.. additional. Nevertheless can you suggest what laptop I should buy? My budget is around 1.2 to 1.6 or 1.7 lakh INR laptop which roughly about 1, 437.14 us dollars to 2, 035.95 United States Dollars. As a professional laptop which one should I buy. windows or MacBook? and which models would it suffice? I'm even ready to wait for the new ai powered chip sets by both m4 MacBooks and xelite windows. Right now I have and radeon hp laptop with 8 gb ram and 1 tb HDD storage with and ryzen processor. What should I do ? Can you help? What and how to manage hpc ? What to buy? My work load generally require 32 gb of ram and graphics of 4060- 4090 nvidia. And I don't know much difference between by how much the processor differ from intel and and but i prefer and. Can you suggest...

yeah minimum requirement say 16 gb or 32 gb ram and vram like 8 gb would suffice. In this course i woul be required to do ml inference and not training. The vram is the limiting factor. And no laptop in this world can do that. Thats why you need data centre and cloud compute.

I need to do basic machine learning inference upto 1-2 billion parameters atleast. And also my course has computer vision, nlp, iot etc.. which has its own requirements....

But the thing here is that in my college they have both data centers as well as cloud set up. And its good.

Why i am asking is that i need the laptop for myself and not to complete 4 years. What should i buy for the best bang to the buck but as well it wouldn't need heavy repairs and it would be reliable through out for atleast 5-6 years.

I understand macbooks soc chips have different efficiency as per what ram to be used and it is same as system. And it is efficient but while talking of windows a lot to be considered as each part means something crucial to the high compute processing and result. Even the overall health.

[tiredofeverythingnew]

Pretty impressive, for the same tests, I was getting around 73 tokens/s on a Windows 11 WSL Ubuntu setup with a RTX 4070 Super GPU (AMD CPU)

[andikunar]

TG largely depends on memory-bandwidth (the SoC has to pump all of the parameters and the KV-caches from RAM into the SoC's caches for each token generated). PP (and ML) is dependent on compute (GPU-horsepower) because token-processing can be batched.

The M4 has 20% faster memory-bandwidth in addition to the faster GPUs. Let's see when Apple will do MacBooks with these chips, maybe I will upgrade my M2. For me, the M3 is not interesting enough for an M2 upgrade.

[leejacksondev]

How did you get a brand new M1 Max in 2024. I’d love one but can only find refurb 😢

[TheStallion]

I want to start experimenting with llms and I have a budget for laptop or pc or a compromise of both, I was going for a great Mac or an ok one and a pc, what’s your advise ?

[TheDanEdwards]

If I was really concerned about performance I would not buy a laptop. An M2 Ultra MacStudio in the refurb store can be had for $3500.

[BrodieChree]

Definitely over value-engineered the M3 Pro, probably using LPDRR5 and PCIe 5.0 clock efficiencies on fewer chips to make-up the differences while increasing profit margins. Also curiously the M3 8gb chips for the Macbook Pro and iPad are the same thing. So also possibly many 8gb Macbook Pros are also possibly iPad chips. Some weird Apple logic going on here. I still think Apple's iPad Pro forecasts for this year were off and they made a product their customers want, a Macbook Pro that does iPad Pro duties.