Run a Local, Private LLM By Downloading Just 1 File - ChatGPT-like Bot on Your PC!

Llamafile lets you run a large language model (LLM) from a single file. It makes LLMs more accessible to both developers and end users. It runs on six OSes on ARM64, AMD64, and Intel x86-64. Just download one file and give it a try!
---

#garyexplains #llamafile

[bazoo]

It is utterly amazing how much of a model can fit in 4GB, together with multi-platform runtime! Excellent work, Llama people, and thanks, Gary!

[GregRutkowski]

Your presentation is the first, I have ever watch, where the chat worked as stated.Many others present chat app's, but fail in the detail of model types, e.g. I run Windows 10 and Ubuntu-Mate. Which by tthe way is the system I'm using to post this comment.

[FlorinArjocu]

A real hit would be to have something like this and then give it access to the internet, so it could search and get up to date information.

[leledumbo]

I just heard about this cosmopolitan in a programming forum, it's quite interesting for a true cross platform executable format, similar to a fat binary but different.

[natjes]

This was very cool!
I´m gonna test this on my M1 Pro 😁
Thanx a bunch!

[pbdivyesh]

Thanks so much, I will run this on my windows machine and apple macbook!
Just waiting for Snapdragon X Elite based windows computers too :)

[cjacobm]

Excellent video! Based on this local installation, is it possible to make API requests, in the same way as chatGPT?

[edwardglynn]

Amazingly easy! HP Chromebook dragonfly in Linux with no issues. Just worked! First thing that EVER just worked in Linux on Chromebook

[xyjldfu]

Hello, is it possible to use Llamafile to "chat" with files (pdf, doc, etc) in a local directory? Thanks.

[ArianeQube]

Any advantage to this than just installing LMStudio and running essentially any model on the huggingface repository?

[asamirid]

neat impressive project i see .. thank you for the instructions and the demo.

[mohammedakbarg]

Thanks Gary for sharing this with us!!! 😀
Really cool

[hilltophorrormoviereviews]

This looks really interesting and I'm gonna try it out. I have two basic questions: 1. Is there such a thing as a context window on locally installed LMMs and if so what is this one's? 2. Can you upload a file (txt, etc) to it that it can read?

[antonydecosta]

Thanks Gary..just tried and it works pretty well.
Is there a way to train this Llm further?

[PaulSpades]

Well, I've tried it. It's rather stupid (not that much you can cram into a 4gb model), but it runs fine and seems to give reasonable responses to anything not too technical or programing related. What's the source of the training data? I'm not at all familiar with these open source llm projects.

[FilmFactry]

What is the use case? Why use this than Bing/Bard/Claude? What can were we have more control?

[TechMeldOfficial]

I am currently downloading it, yet to test it !

[Dhruv_s]

Anyone please give me simple instructions that how to use this LLM with GPU as well.

[rgalik]

Interesting, I wonder about running it (natively) on Android - which is Linux under the hood...

[Riggzilla]

Is there a guide to run the llamafile with your GPU? I'm using GPU passthrough on a VM.