Graph RAG with Ollama - Save $$$ with Local LLMs

Learn how to set up a local model using OLAMA and the Groq API for Project GraphRAG. We will also look at the limitation of using smaller local open weight models.

LINKS:

💻 RAG Beyond Basics Course:

Let's Connect:

Signup for Newsletter, localgpt:

TIMESTAMPS
00:00 Introduction to Project GraphRAG
00:30 Setting Up OLAMA and Choosing Models
01:14 Configuring GraphRAG with OLAMA
02:32 Using Groq API with GraphRAG
04:35 Running Local Indexing and Entity Extraction
07:15 Testing and Comparing Models
08:15 Importance of LLM Choice in GraphRAG
10:58 Optimizing Prompts for Different LLMs

All Interesting Videos:

[programwithpradhan]

Definitely waiting for fully customisation of graph RAG using open source models

[JohnBoen]

Great discussion.

I am tackling things differently, and it seems to work pretty well.

I use ChatGPT to manually construct subject-predicate-target statements from a document. Instructions to infer root names from prepositional references, etc...

I feed this into a graph database.

User-entered text is passed to a local Llama3 instance to construct a graph query - this query is executed against the graph database.

This result set is added to the initial user text and passed to an appropriate LLM.

[maxs]

Hey! Cool video. I actually built a full local solution using Ollama, no need for LM Studio at all. Here's what I did: I created a proxy that translates between OpenAI API embeddings and Ollama's format, both ways.

The cool thing is, it works flawlessly for both global and local queries. I'd be happy to share the script with you if you're interested!

[maxs]

ollama embeddings proxy on git hub. This script bridges the gap between OpenAI's embedding API and Ollama, making it compatible with the current version of Graphrag

[MeinDeutschkurs]

Yeah! Prompting is key. Llama3 is very good at step by step instructions. Print this, write that, do this and combine 12:08 that, finally this. The good thing is that larger models are also able to understand this, but most of the time it does not work in the other direction.

[jasmeetsingh]

Hey, great video, A couple questions

1. Can I create the Graph using Llama3-70B and then use a different LLM (which doesn't have a rate limit) to answer RAG queries ?
2. Can I create a partial Graph, and update it as more data comes in (rather than create the Graph all over again) ?

[BrandonFoltz]

Yea I tried using llama-3 in LM Studio and using OpenAI embeddings since its cheap. GraphRAG detonated (after waiting an hour of course). It seems like it did all the LLM stuff OK and embeddings OK, but at the end when trying to put everything together it just went to crap. Too specific to using OpenAI for everything. Even using 4o it was more expensive than a six pack of beer and I ain't giving that up.

[nguyenanhnguyen]

May work with English-only and not even work with vLLM due to buggy parameters !

[MrAhsan]

Does this RAG provides better results compared to semantic chunking?

[awakenwithoutcoffee]

what about Lamini fine-tuning ? this might just be the best of both worlds. Would be really interesting to see comparisons between traditional RAG (optimized techniques), graphRAG and fine-tuning (lamini).

[shameekm]

For embedding model as well can't we use ollama? I see that mixedbread mxbai is available.

[meelanc]

Thanks for the video, but are you, by chance, sharing the code modifications? I did not see any links.

[mrchongnoi]

Definitely have to use this with local models. I burned $46.00 in tokens. I waited 10 minutes before I could test the application. Your text files have to be 8-bit clean, or the app will blow up. Graph Rag is not a gift from microsoft.

[Sceptic]

Could you use gemini Flash or Claude Haiku or DeepSpeed V2 to keep costs down??

[takshitmathur]

if my data size is larger(say 1000 pdfs) then the embedding cost will be too high, even if we use local models the time taken is too high right now. what do you think about using gemini pro model for doing that, as google charges no cost up to 300$ for AI projects. Maybe you can suggest your views and make a video on this in future?

[TranKiet-pjmw]

actually, when going with entity extraction, people can extract Noun and adj as a text fiile then make a cluster, using it in struct to send a request to bigger model such as chat GPT , the struct is well defined , accoding 5 topic of science so it always return a value you want to looking and decreasing the complexity . my structure is simple , define , formulate, it s shortcut of which ? ( looking for enpoint ), middle point ( by tracking respond then sending again), .. the structure will depend on user . Let define "Noun", is that Noun using for approximating a thing as a word ? . so when we now Noun and define for that word, we know the relevant topic around it , and because synonym is limited, so just respond over time , it would get the converge information, and luckily , we could use a little trick to get 1000 account for free . but my method is still limit when i dont define math formulate, and other language except english, some define still not unclear enough and fragment information only sense when take big sample .
your method is great too, why have to stick in one idea, the more would be better . thanks

[brucewayne]

Have you tried to use it with an existing neo4j graph ?

[xinzhang]

So compared to GPTs, his search generation effect will be better?

[readmarketings]

If Graphrag can’t match the quality of Diffbot while being cheaper, it’s currently not useful.

[lesptitsoiseaux]

Doesn't work on PC. I'll try on Mac. I haven't found a video that actually makes it work on PC locally.