All You Need To Know About Running LLMs Locally

RTX4080 SUPER giveaway!
Please read all the rules & steps carefully!!
1. Sign-up for NVIDIA's Virtual GTC2024 session between Mar 18 - 21st
2. Participate the giveaway DURING Mar 18 - 21st
3. ???
4. Profit

TensorRT LLM

Chat with RTX

Links:

Model recommendations:

This video is supported by the kind Patrons & YouTube Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon

[Music] massobeats - magic carousel
[Video Editor] maikadihaika

[bycloudAI]

Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML. (thanks to danielmadstv)

[ambinintsoahasina]

The amount of infos you give both in the videos and the descriptions is insane dude! Keep up the good work!

[danielmadstv]

Thanks for the video! Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML.

[noobicorn_gamer]

I hoooonestly don't know how to feel about the thumbnails looking too similar to you-know-who that got me accidentally clicking this video but meh... One's gotta do what one's gotta do I guess.

[lunadelinte]

that was awesome, thanks for the concise information bycloud! 🔥

[papakamirneron]

Immensely helpful video. I hope the future has tonnes of user controlled locally ran llms for us in store!

[flexoo]

Poor Faraday nearly always gets overlooked when people talk about local LLMs, but it is without a doubt the most easy to use "install and run" solution. Unlike nearly all other options it's near-impossible to mess something up and default settings out of the box are not sub-par.

[RetroPolly]

A thousand thanks! Finding a good LLM model was a complete nightmare for me + it is difficult to figure out which formats is outdated and which - new hot stuff.

[johnsarvosky]

Thanks, this is great. Please make a comprehensive video on Fine-tuning locally 101..Cheers

[robertmazurowski]

I was pretty sure this was a fireship video, but the video is great and informative. Exacly what I was looking for.

[joseph-ianex]

with local models are you able to make much longer responses given that you have enough ram and vram?

[lintalyor]

I like this simple explanation with the video editing thanks!

[juanantonionieblafigueroa]

Your videos are way more fun than my algebra homework

[vladislava]

Very nice, tons of useful info
Thank you!

[aketo]

Thank you. Very interessting. Is it possible in LM Studio to work with own files? Or create own LLM or extend LLM for own cases?

[Hkari_]

I love your adhd-friendly edits cloudy. <3

[kernsanders]

In regards to context, would LLM Lora's help with that? Lets say im busy with story writer LLM and the fantasy world I'm working with would be as big as something like Middle Earth from LOTRs. Would a Lora help with that? Like if I train a Lora on all our past chat history about the story etc. Also more text regarding the lore of places and history of characters and family trees. So taking that into consideration, would that assist in keeping the context low? So I don't need to keep a detailed summerized chat history etc. What would the requirement be for training such a Lora and what would the minimum text dataset require for a coherent training?

[christopheralvarez]

Where do I upload the photo once GTC comes around ?

[trolik]

Absolutely fantastic and informative video. Well done! I will say I feel like the information certainly speaks to the grip that OpenAI has, especially from a development standpoint, despite the whole video being about open-source models.

The procedures, time, research, and money required for any rando or small (even mid size) business owners to integrate open-source and local AI without any practical knowledge about it is near impossible. OpenAI wraps up RAG, "fine-tuning", and memory nice and neat into Assistants which can be easily called via the API. It would be amazing to have a completely standardized system that allows for the same type of application, but geared towards the variety of open-source models out there. Some platforms like NatDev let you compare multiple models based on the same input. Being able to see how RAG and fine tuning affects different models, both open-source and non, from the same platform would be unreal.

[Leo_Aqua]

You can also use ollama. It even runs on a raspberry pi 5 (although slow)