Run ALL Your AI Locally in Minutes (LLMs, RAG, and more)

Update! Follow up video for deploying this app to the cloud!

Artificial Intelligence is no doubt the future of not just software development but the whole world. And I'm on a mission to master it - focusing first on mastering AI Agents.

In this video I show you an INCREDIBLE packaged local AI solution with Ollama for the LLMs, Qdrant for RAG, Postgres for the SQL database, and n8n for the no code workflow automations. This package is SO easy to set up and use! I walk you through the setup quick and then even show you how to use it to build a RAG AI Agent that runs entirely locally!

One really important thing to keep in mind is that while I created a no code AI agent with n8n in this video, this whole local AI setup could be used to actually code your AI agents! It doesn't have to be "no code" n8n for the actual agent. A custom coded RAG AI agent with n8n for the accompanying workflow automations (agent tools!) is another fantastic use of this local AI package.

Hardware: I am actually using a laptop for this video - with a 3070 laptop GPU and an i7-12700H CPU with 14 cores and 20 threads. So nothing crazy at all! It takes up to 5 minutes for me to get a response from Llama 3.1 8b including RAG. Definitely not fast enough for a chatbot though those kind of speeds could be okay for some agentic workloads. Typically though you will want a GPU with at least 8GB of VRAM for an 8b model, and at least 16GB of VRAM for a 70b model like Llama 3.1 70b.

00:00 - 01:17 - Unveiling the Local AI Package
01:18 - 06:38 - Installing the Local AI Package
06:39 - 07:45 - Exploring the Docker Containers
07:46 - 18:07 - Creating a Local RAG AI Agent
18:08 - 19:18 - Testing the Local RAG AI Agent
19:19 - 20:19 - Outro + Future Ideas

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

My version of the local AI starter kit with my improvements, as well as the template for the n8n RAG AI Agent workflow I created in the video, can be found here:

Link to the Local AI Starter Kit by n8n:

I recommended GitHub Desktop and Docker Desktop in the video - here are the links for both:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Join me as I push the limits of what is possible with AI. I'll be uploading videos twice a week - Sundays and Wednesdays at 7:00 PM CDT!

[ColeMedin]

Follow up video here for deploying this to the cloud!

[dineshbabu]

This is a very good step-by-step tutorial. Following the instructions in this video will get you started with local AI. For people trying M1 and above, the ollama must be installed separately, and the rest are the same.

[ifnotnowwhen]

Cole, you’ve done an outstanding job! Your videos consistently make complex topics clear and easy to follow, which is a rare talent. I appreciate how you break down each concept, making it accessible for viewers at all levels. Your teaching style not only engages but also inspires confidence in learning.
I’m eagerly anticipating your next iteration. It’s always exciting to see how you evolve your content and introduce new ideas. Keep up the fantastic work, and thank you for sharing your knowledge with us!

[datpspguy]

By far the best tutorial and overview on Local RAG and also dropping gems on the little improvements you've made from the original repo. Workflow is amazing too!! One of my ideas is playing some of older rpg's back in the day on the steam deck but with less time that I have now for other priorities, its nice to just query the walkthrough docs and ask where to go next etc.

[jordon]

man.. just dropping casual double entendres as hole references? that’s an instant sub

[Hydrawindforce]

This is the best example I have seen for the Local AI Agent and Rag

[isohumulone]

Local is a good start. As a practical application, I think a good project would be to distribute the containers and have a login system for businesses.

[LuisYax]

The removing of the vectors records, when reimporting and updated file fixed a lot of my problems. Thanks for the help. U da man!

[nwokobia]

Outstanding work, Cole. Love it. I will implement it today. Looking forward to more of your excellent content. You are not verbose, just straight to the point and deliver value to boot. Thank you!

[rbp]

Genius, this is like a “medior ai engineer” tutorial video if someone builds the same thing then tweaks it to make a unique llm app out of it. I think a lot of companies would appreciate their engineers to know all this

[michamohe]

Something I'd like to see is building in mixture of agent frame works and tool use and an actual chat interface. This is a great start and that is exactly what I'm going to start working on lol

[Joooooooooooosh]

Open Web UI is still the best and cleanest implementation I've seen.

[alexlanders]

I'm excited to see you extend this! Working Supabase into this flow for authentication, etc would be incredible. Awesome video bro!

[damadorpl]

You can use also Postgress with pgvector instead of Qdrant

[HermanRas]

Thank you sooo much.. I had to change the document import a bit to work with local network shares for my company but it works .. SO GOOD.

The deleting of documents already in the system before adding more is really important, ** I cant wait for your front end video **

[bakuleshsuhasrane]

Outstanding Bro I was looking for this solution !!!! since long months

[jimbob]

Thank you for reminding me of this! Keep to this type of content for the people who want to benefit with our own offline AI ventures!

[fronix]

Should be illegal to use github in lightmode

[warfy]

Thank you so much for this. It's exactly what I was looking for. The n8n data flow is just so useful. I've been looking at how to make an AI for a finance customer, and they needed private data injested so retrieve information. Just need to combine with a voice assistant to have a completly internal ai assistant.

[henrismith]

I'm excited. I just bought my first desktop PC and went all out with a 4080 Super and a Ryzen 7 7800X3D. I managed to pick up a mint condition use Samsung Odyssey CRG9 2x Quad HD monitor so gaming will be fun, but I'm equally excited about playing around with some AI stuff. I love the idea of fine tuning models, using RAG, oh and especially stable diffusion. I keep finding out things about my Nvidia GPU that I didn't know, like being able to watch everything in HDR which is fantastic because so little content is in HDR but it looks so good.