Host your own LLM in 5 minutes on runpod, and setup APi endpoint for it.

Показать описание

Please note if you are using this for anything other than testing you should restrict access with an APi key.

Thomas Hill

Рекомендации по теме

Комментарии

I couldn't run it the whole day yesterday, your steps by steps approach is wonderful. Thank you.

kaynkayn

I was really looking for something like this. Thank you so much. Can u make a video on how to use Agentkit by BCG

prnmdid

Hi Thomas, can you provide guidance on how we select the GPU based on the model we would like to test? For example, if I want to test Goliath 120b at reasonable speeds, how do I know which GPUs to deploy? Thanks.

nxuhdbg

Which is the most cost effective way to host our LLMs on Runpod, using serverless or using a runpod?
Use case: no production level, just testing different LLMs, even in some autonomous agent networks, which can burn money pretty quickly using gpt-4, so running local LLMs on Runpod some times a day for some hours, should not be always on and instance should not spin up very quickly...
I think serverless is the better for this use case, but I'm not sure, so what is your opinion?

attilavass

Is this not a bit slow for a 7B model running on a (freakin!) H100 ?
Getting roughly same speed here with a RTX2070 and 5bit quantized 7B models ...

Thanks for the tutorial tho, was gonna look into runpod.

nemai

I keep getting HTTP service not ready for the ports. Is there an additional step required for this?

kelv

i cant connect to http port 7860. it says its not ready. and also on the logs i am getting this error "AttributeError: module 'gradio.layouts' has no attribute '__all__". can you help please.

emiryuce

im getting a 405. I dont think i used the blokes template with API also on thats why I guess.

obygknu

Host your own LLM in 5 minutes on runpod, and setup APi endpoint for it.

Run Your Own LLM Locally: LLaMa, Mistral & More

API For Open-Source Models 🔥 Easily Build With ANY Open-Source LLM

host ALL your AI locally

This new AI is powerful and uncensored… Let’s run it

How to deploy LLMs (Large Language Models) as APIs using Hugging Face + AWS

Running a Hugging Face LLM on your laptop

All You Need To Know About Running LLMs Locally

Run your own AI (but private)

Taqelah Meetup July 2024 - Building and Testing LLM Apps with RAG - Srinivasan Sekar

Build a Large Language Model AI Chatbot using Retrieval Augmented Generation

Create a Large Language Model from Scratch with Python – Tutorial

Should You Use Open Source Large Language Models?

Host your own LLM in 5 minutes on runpod, and setup APi endpoint for it.

Training Your Own AI Model Is Not As Hard As You (Probably) Think

Build your own LLM AI on a Raspberry Pi

How to Fine-Tune and Train LLMs With Your Own Data EASILY and FAST- GPT-LLM-Trainer

Locally-hosted, offline LLM w/LlamaIndex + OPT (open source, instruction-tuning LLM)

Create Your Own LLM from Scratch Easily

Run Your Own ChatGPT-like LLM on Your Windows PC!

Deploy and Use any Open Source LLMs using RunPod

Deploy FULLY PRIVATE & FAST LLM Chatbots! (Local + Production)

Run ANY Open-Source Model LOCALLY (LM Studio Tutorial)

Replace Github Copilot with a Local LLM

Using ChatGPT with YOUR OWN Data. This is magical. (LangChain OpenAI API)