How to Run LLAMA 3 on your PC or Raspberry Pi 5

Meta (i.e. Facebook) has released Llama 3, its latest Large Language Model. It comes in two sizes, an 8 billion parameter model and a 70 billion parameter model. In this video I look at how you can run the 8 billion parameter model on your PC or even on a Raspberry Pi 5.
---

#garyexplains

[sonofabippi]

This was such a pleasant, easy to digest video about llama 3, that the next video started playing and I was like, "wait, that dude was awesome. I need to go back, do all the things (like, subscribe, bell)!"

[coolParadigmes]

Amazing that such a huge LLM can run on a Raspberry PI 5! By the way for the towel question (3 towels=3 hours to dry, how long for 9 towels?) it describes correctly the reasoning to use [it depends on the towels characteristics (supposed identical) and the drying process], but it gets it wrong anyway 🙂 1 hour/towel which is only valid with a dryer paradigm with a drying power of precisely one towel/hour, but in the sun and plenty of space it takes the same time independently of the number of towel, so 3 hours.

[maxdiamond]

Thanks Gary great video, nice and direct. Exactly whats needed to get up and going, can't wait to test this.

[EddieLongStockings]

"What is my purpose?"

"You pass the butter."

[peterfrisch]

Gary, would a Raspberry Pi run Llama 3 faster with a Google Coral TPU?

[andueskitzoidneversolo]

never heard of the towel question. i like it. like asking if a pound of feathers on Jupiter also weighs the same on earth.

[chrisarmstrong]

Did it give the correct answer for the towels question ? It looked like it was about to say 9 hours.

[mikeburke]

Thanks, definitely going to give it a try.

[JNET_Reloaded]

ollama run dolphin-llama3 <<< uncensored version!

[iAmMeAndYouAreYou]

Did you speed the reply up? I have the phi 3 mini on my rpi5 (4gb) and its insanely slow.

[ManthaarJanyaro]

Can I feed the data to it, and ask from that Data?

[MRLPZ]

Interesting video, any experience with other models, maybe SLMs like Phi-3 on raspberry pi. Also on smaller Raspis. Is there a chance to run this on a Raspi Zero 2 ?

[xuldevelopers]

When recommending software to install on users' computers, how do you verify its source? Have you read the terms and conditions? What do you know about that company?

[GGUO]

Are those examples using a quantized version of the model?

[MrStevemur]

Sounds fun, but I'd like it to be able to analyze input from sources other than what I type into a chat window. E.G., "Read report_final.docx and generate a summary." Are we there yet?

[exuply]

hey which ram version of raspberry Pi u are using?

[timr.]

What would be the specs required for the 70 billion parameter model?

[Wobbotherd]

Could a Mac with sufficient RAM run the full model?

[vasudevmenon]

Any reason why llama file wasn't used since you can swap out gguf file which works on nvidia, m1 and experimental support for rocm?
Duck duck go AI chat is quite good and didn't need login. Claude felt a bit more natural than openai gpt4.
Is llama3 a new foundation model that can perform on par with higher parameter in condensed form?

[PrashantKumarPrasad]

Can you share codes to run llama3 in raspberry. I don't think it is possible to run llama in raspberry !