LLMs with 8GB / 16GB

Can a modern LLM like llama 2 and llama 3 run on older MacBooks like MacBook Air M1, M2, and Intel Core i5? Sort of and i depends on which model.

Use COUPON: ZISKIND10

🛒 Gear Links 🛒

🎥 Related Videos 🎥

— — — — — — — — —

❤️ SUBSCRIBE TO MY YOUTUBE CHANNEL 📺

— — — — — — — — —

Join this channel to get access to perks:

— — — — — — — — —

#machinelearning #llm #softwaredevelopment

[TechGameDev]

10:05 I believe that, for machine learning, it uses VRAM. On Intel Macs, it does not use unified memory and does not share RAM with the graphics card, as the graphics card has its own dedicated RAM called VRAM. In contrast, Mac Silicon shares RAM with the graphics card, making the GPU's RAM demand almost unlimited.

[QuantumCanvas]

Kind of stuff I was searching for. Thanks Alex

[burprobrox]

I had a powermac in the 90s with 16 slots for ram, my last new Mac was the first gen Air, and a core duo mini. I was working for Apple at the time and got a crazy discount. I really miss the old days and Jobs was the best boss ever. I’ll never forget his goodbye email to employees, we were literally all tearing up. Feels like a different universe since then.

[SvenReinck]

Q4 means the weights of the model are saved as 4 bits. The original is in FB16 which is floating point numbers with 16 bits.

[dmug]

I compared a M1 Mini vs a 2013 Mac Pro, and one of the tests I did was with Ollama. It was one of the very few tests that the Mac Pro 2013 had the clear advantage thanks to the 64 GB of ram

[sarjannarwan]

6:23 it isn’t that they’re trained on more data. At the start of training weights and biases will be initialised, they’re just altered during training. The difference would be in the architecture.

[aflury]

Quantized models are trained on less data? I thought they were just reduced precision representing the same training. Like turning up lossy compression, it gets pixelated.

[nommchompsky]

Me and my 16gb M1 Air are thankful for this video

[propavangameryt]

0:20 😅 i still own a MacBook Air 2015 core i5 model, it still works perfectly fine for regular browsing watching movies & stuff but 😂 I don’t have to keep it plugged omg

[ronanpelodefuego]

I would like to run some llm in local for creating video, changing voice, etc. I’m thinking in buying MacBook Pro m4 pro 48 Gb. Would it be enough? Thank you very much!

[xCUBE]

great videos! You should do a video comparing the various 7B-16B models

[RichWithTech]

Can you do more amd / Apple arm/ snapdragon Comparisons pls

[miacodesswift]

I’ll try on a mid 2020 macbook air with a 5700XT egpu

[tutacat]

If your model is larger than your memory, it has to load each part separately for every inference step, since the whole model needs that info. VRAM is not separate unless you have a dedicated GPU, which most Intel MacBook Air's do not.

[halycano]

Since you have an older Mac, it would be interesting to see trying to do modern dev work on these older unsupported Macs. If you could do it on a Mac that used OCLP, that would probably be a more interesting video.

[whoadog]

I have a complete oddball m2 Mac Mini with 24gb of ram that I got as a refurb from Apple. I need to try some of the new models out.

[lalitsharma]

Please do a Mac mini review when it gets upgraded.

[tutacat]

Actually q4 means 4-bit quantised. The original models are usually 32-bit, so that's 8x smaller.

[peterihimire]

Is 8gb RAM enough in 2024? Apple Yes, others No.

[gustavo]

Should have mentioned to avoid quants that don’t have k_m K_s or _x, q4_00 for example is worse and slower than q3_xs