filmov
tv
Install and run LLM's Locally with text generation webui on AMD gpu's!
Показать описание
Let's setup and run large language models similar to ChatGPT locally on our AMD gpu's!
### Installing ROCm
sudo apt update
sudo apt install git python3-pip python3-venv python3-dev libstdc++-12-dev
sudo apt update
sudo amdgpu-install --usecase=graphics,rocm
sudo usermod -aG video $USER
sudo usermod -aG render $USER
sudo reboot
### Installing Text Generation Webui
mkdir ~/gpt
cd gpt/
cd text-generation-webui
# Setup virtual env
python3 -m venv venv
source venv/bin/activate
# Install correct torch ROCm5.6
# Install the rest of dependencies
# Installing with no avx support
# Create launch script
# Inside launch script paste:
#!/bin/bash
source venv/bin/activate
export HIP_VISIBLE_DEVICES=0
export HSA_OVERRIDE_GFX_VERSION=11.0.0
### save and exit your launch script
# Make script executable
# Now you can launch webui with your script
Model from the video:
TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ
Some settings to check if models are not loading:
Disable exllama
float 32
fp4
Transformers works most of the time, not always performant though.
Generally load_in_8bit and load_in_4bit will not work -- it uses bitsandbytes which has bad ROCm support.
### Installing ROCm
sudo apt update
sudo apt install git python3-pip python3-venv python3-dev libstdc++-12-dev
sudo apt update
sudo amdgpu-install --usecase=graphics,rocm
sudo usermod -aG video $USER
sudo usermod -aG render $USER
sudo reboot
### Installing Text Generation Webui
mkdir ~/gpt
cd gpt/
cd text-generation-webui
# Setup virtual env
python3 -m venv venv
source venv/bin/activate
# Install correct torch ROCm5.6
# Install the rest of dependencies
# Installing with no avx support
# Create launch script
# Inside launch script paste:
#!/bin/bash
source venv/bin/activate
export HIP_VISIBLE_DEVICES=0
export HSA_OVERRIDE_GFX_VERSION=11.0.0
### save and exit your launch script
# Make script executable
# Now you can launch webui with your script
Model from the video:
TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ
Some settings to check if models are not loading:
Disable exllama
float 32
fp4
Transformers works most of the time, not always performant though.
Generally load_in_8bit and load_in_4bit will not work -- it uses bitsandbytes which has bad ROCm support.
0:06:55
Run Your Own LLM Locally: LLaMa, Mistral & More
0:10:30
All You Need To Know About Running LLMs Locally
0:08:52
How To Install Any LLM Locally! Open WebUI (Ollama) - SUPER EASY!
0:14:11
Run ANY Open-Source LLM Locally (No-Code LMStudio Tutorial)
0:06:06
Ollama: Run LLMs Locally On Your Computer (Fast and Easy)
0:06:31
Ollama on Windows | Run LLMs locally 🔥
0:14:21
Install and Run Llama 3.1 LLM Locally in Python and Windows Using Ollama
0:11:12
How To Run ANY Open Source LLM LOCALLY In Linux
0:18:02
3. CREATING A LLM CHATBOT USING STREAMLIT 😍🫠
0:04:31
Run LLMs locally - 5 Must-Know Frameworks!
0:06:53
How To Run LLM Locally on Any Computer With LM Studio (LLaMa, Mistral & More)
0:04:35
Running a Hugging Face LLM on your laptop
0:15:09
FREE Local LLMs on Apple Silicon | FAST!
0:09:30
Using Ollama to Run Local LLMs on the Raspberry Pi 5
0:14:50
Run Your Own ChatGPT-like LLM on Your Windows PC!
0:10:59
How to Install TextGen WebUI - Install ANY LLMs IN Minutes Locally! (Oobabooga)
0:09:07
Run LLMs without GPUs | local-llm
0:06:02
Ollama: The Easiest Way to RUN LLMs Locally
0:09:45
AI Unleashed: Install and Use Local LLMs with Ollama – ChatGPT on Steroids! (FREE)
0:01:11
How to Install AI Locally: Step-by-Step Guide to Running LLMs Using Jan.AI
0:06:56
RUN LLMs Locally On ANDROID: LlaMa3, Gemma & More
0:08:36
AI Toolkit for VScode - Install Locally - Run LLMs Locally in Windows
0:03:41
Ollama, How to Install & Run LLM's on Windows in Minutes!
0:13:21
Your Ultimate AI Copilot on the Desktop ! Run ANY LLMs Locally
Комментарии