How to run a local AI chatbot on Windows in 5 min, no cuts, no edits, with Ollama, LMStudio, OpenAI

Running a local AI chatbot on Windows in just 5 min, no cuts, no edits, with Ollama and LMStudio

[CodingAdventures]

I like your experiments Scott... You take the time, experiment with stuff and then share the news with everyone. These local model experiments are also very cool!

[PinheiroJaime]

Nice video, Scott. I have both Ollama and LM Studio running in my laptop. Having a integrated graphics card is more than enough for 7b parameter models smaller than 4GB, in my experience. The output is not that fast, still faster than I can read.
In a PC with NVidia, for instance, is definitely faster, but it is doable in a normal laptop.

[Frostbain]

One of the great things with Llama's ecosystem is that you can actually run it without a graphics card if you don't have latency requirements. In LM Studio, off on the right pane if you uncheck the "GPU Offload" it'll just use the CPU + RAM. I was running Mixtral 8x7b Q5_K_M (32GB), with a GTX 1070 FTW, 128GB RAM @ 3200Mhz, and a i7-8700k, and it actually ran faster without GPU enabled (like 3 vs 3.5 tokens / second).

Might be interesting to go over Autogen too. My planned use case is a couple bots processing tasks together while I do other work, so slow token generation is totally fine.

[webluke]

The "uncensored" models seem to give the best results. No bias training applied and not refusing to give factual results.

[jazzweather]

Wow, I didn't know Scott Hanselman had a YT channel... instant subscribe.

[jayhu]

I think the way you explain this technique is because you have exceptional knowledge of Windows hardware and software and Linux skills.
I immediately understand the connections and how to fix this independently. If possible, more AI hardware, compute tips to run various LLMs locally.
Thank you for the explanation

[rabidtommy]

Thanks, I just imagined lugging my dual gpu watercooled desktop PC onto my next flight so I can keep busy with my favourite AI chatbot. Hope I can get it through security!

[mtranchi]

The audio could use a little boost. Thanks for the vid!

[JosuaBatubara]

Thanks for the tutorial, Scott! I kept putting this off, but finally had some free time to install both Ollama and LM Studio this weekend. My aging computer struggles with it, but it's still workable. I guess it's a good excuse to buy a new one 🤣🤣

[Siderite]

Eject a tape from a VCR, if you're old. You crack me up!

[DiegoAguilera]

Thank you Scott! Would love to see more videos related to running models locally

[zandorachan]

very helpful video, thank you! 'hallucinate' is DEFinitely a more fun term to describe weird model responses, than 'not grounded in reality' ;)

[rudyMents]

If anyone stumbles on this with a more recent version of LM Studio, the GPU Acceleration option is now inside the Advanced Settings section in the right panel. Instead of typing in "-1" you can just click the [max] button.

[zbart]

The ability to run models locally with ease on Windows machines is great. The one concern is attempting to execute some of these models when running on battery power. Plan on your battery dying quickly.

[dlonicholas]

This is sooo cool. Makes me want to get a machine with a dedicated gpu.

[coderider]

Good video, perfect for when your security team have locked down the azure open ai resource group and no one can do prototypes.

[EricRohlfs]

I'm running lmstudio on $200 laptop with 32GB ram. Crappy integrated video card. It runs. Takes 20 seconds to answer... but can be done. Im not doing anything special lmstudio out of the box. GPU better, no doubt.

[TheVideoGameVault]

Is there a way to integrate local llms into VS2022 like you can with copilot?

[darkenaxe]

This is so insanely accessible, i had no idea !

[ryanoc]

Great job showing how easy it can be. Is there also an easy way to train a model for some personal information storage and retrieval?