I Ran ChatGPT on a Raspberry Pi Locally!

Product Links (some are affiliate links)

Here are the instructions to run a ChatGPT-like model locally on your device:

4. You can now type to the AI in the terminal and it will reply.

If you prefer building from source, follow these instructions:

For MacOS and Linux:

3. Run `make chat`.
4. Run `./chat` in the terminal.

3. You can now type to the AI in the terminal and it will reply.

As part of Meta’s commitment to open science, today we are publicly releasing LLaMA (Large Language Model Meta AI), a state-of-the-art foundational large language model designed to help researchers advance their work in this subfield of AI. Smaller, more performant models such as LLaMA enable others in the research community who don’t have access to large amounts of infrastructure to study these models, further democratizing access in this important, fast-changing field.
making LLaMA available at several sizes (7B, 13B, 33B, and 65B parameters) and also sharing a LLaMA model card that details how we built the model in keeping with our approach to Responsible AI practices.

Over the last year, large language models — natural language processing (NLP) systems with billions of parameters — have shown new capabilities to generate creative text, solve mathematical theorems, predict protein structures, answer reading comprehension questions, and more. They are one of the clearest cases of the substantial potential benefits AI can offer at scale to billions of people.

Smaller models trained on more tokens — which are pieces of words — are easier to retrain and fine-tune for specific potential product use cases. We trained LLaMA 65B and LLaMA 33B on 1.4 trillion tokens. Our smallest model, LLaMA 7B, is trained on one trillion tokens.

Like other large language models, LLaMA works by taking a sequence of words as an input and predicts a next word to recursively generate text. To train our model, we chose text from the 20 languages with the most speakers, focusing on those with Latin and Cyrillic alphabets.

There is still more research that needs to be done to address the risks of bias, toxic comments, and hallucinations in large language models. Like other models, LLaMA shares these challenges. As a foundation model, LLaMA is designed to be versatile and can be applied to many different use cases, versus a fine-tuned model that is designed for a specific task. By sharing the code for LLaMA, other researchers can more easily test new approaches to limiting or eliminating these problems in large language models. We also provide in the paper a set of evaluations on benchmarks evaluating model biases and toxicity to show the model’s limitations and to support further research in this crucial area.

To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. Access to the model will be granted on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world. People interested in applying for access can find the link to the application in our research paper.

We believe that the entire AI community — academic researchers, civil society, policymakers, and industry — must work together to develop clear guidelines around responsible AI in general and responsible large language models in particular. We look forward to seeing what the community can learn — and eventually build — using LLaMA.

[aaronjennings]

Quantization, in plain English, is a process of representing something in a simplified or discrete form. It involves reducing the complexity or precision of something to make it easier to work with or understand.

Think of it like taking a detailed painting and converting it into a pixelated image. Instead of having many different shades and colors, the pixelated image uses a limited number of colors or pixels to represent the overall image. This simplification makes it easier to store, transmit, or process the image.

In the context of data or numbers, quantization involves reducing the number of possible values or levels that can be used to represent a measurement or a quantity. For example, instead of representing a measurement with infinite decimal places, quantization rounds it to a specific level of precision, such as rounding a decimal to the nearest whole number or a certain number of decimal places.

Quantization is commonly used in various fields, including digital signal processing, image and video compression, and data storage. It allows for more efficient use of resources, faster computations, and simpler representations, while still preserving the essential information or characteristics of the original data.

[szmonszmon]

So, you don't know what you run? "I ran ChatGPT". I skipped some parts of video to see what you really have and I saw llama and alpaca. I was really curious where did you found the ChatGPT source code... No, llama, alpaca and others are not the same as ChatGPT. They don't understand other languages than english and they have issues with other code languages than python. So, im some circumstances they are similar to ChatGPT, but only in certain use cases...

[rodrigo_plp]

So true. Many models have heavy requirement to run, like 16 GB of RAM, but depending on your use case you can get away with a lot less. I got surprising results using a vector database and Llama 2 even with 8 GB of RAM and 4 CPUs. In Supawiki (disclosure: built by me) I am using a bit more than that, and the results are impressive. Exciting stuff indeed.

[polloman]

I'm the only one that feels a little bit of nostalgia realizing that the world where I grew up in is already gone? When my parents were young, they had these room sized computers. My mother used to be a typewriter secretary. My father used to be a mechanic back when cars were carbureted at around high school. When I was a child, maybe 4 years old, I remember my father had a thiccc IBM laptop from work. Our first digital camera had only 256MB of memory. Today we're running AI models in a computer a little bit bigger than a wallet. I can only imagine what is waiting for us in a couple of years. Life's good :)

[DiscontinuedRBASIDOWK]

Great video, a little misleading to call it ChatGPT considering the power of ChatGPT compared to this much smaller model but still a great video. Well done.

[wood]

That is impressive. I'm able to run 7B Q6 models on my old pc with an RX 580 and small language models like Phi 2 runs faster that I can read. I believe the future of LLMs is gonna be local instead of cloud due to privacy as you said.

[Vypertech]

2040: i created a new universe using phone parts

[alexiscolonfpv]

hi i have a error on my PI
main: seed = 1712111644

llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 6065.34 MB
Segmentation fault
can you help
thanks

[OblivifrekTV]

Wonder if it would work better with a Raspberry Pi Cluster

[MorrWorm]

Yo, I took a Pi 4 (8gb) with argon 40 case that has the m2 put a (1TB) SSD added Ubuntu and I love it! Loads fast responsive. I have my M1 MacBook, my Mac mini & a older HP running windows. Now I have a Linux desktop. I know I could have ran a VM. I enjoy bare metal. Great video. Liked & subscribed

[DComputing]

Thank you, I knew it was small I didnt realise just how small. 10/10 short sweet concise.

[OVERLOARD]

I am using big LLMs on a bunch of Tesla p40s but since the cooling options are pretty loud and it consumes a lot of energy I wonder if I get better inference with a coralAI TPU on a raspberry pi than using llms on the pi without anything else. Also, would it make sense to build a pi cluster, each fitted with a coral ai tpu via pcie port?

[NicksonNg]

I tried running on a pi 5, and its still not very usable even tho theres performance boost

[Mauroplcr]

Hi, nice tuto, but dosent work to i have only 4gb of ram llama_model_load: ggml ctx size = 6065.34 MB
Segmentation fault (core dumped)

[onghiem]

can you integrate USB Coral AI accelerator to make this RPI faster, or could you run this on a PI cluster?

[shiftednrifted]

did you run chatgpt? or did you run one of them broke ass local llms that lose the thread on conversations close to immediately, run out of tokens way too fast to be useful for most workloads, take forever to infer even on the highest end consumer hardware, and otherwise don't even slightly compare to chatgpt.

its cool you ran it on a raspberry pi, but it is NOT comparable.

[gn]

I'm eagerly waiting to see the video on running the model on the Jetson Nano!

[mrguiltyfool]

how do you get tts to read out the chatbot output

[georgeshafik]

Great video, simple steps to follow everything worked the first time. It was slow and I used identical hardware to yours. Really interested in using a larger Lama model with Nvidia Jetson👍

[StephenBrown]

Your passion for this stuff is magnetic!