Gemma 2 - Local RAG with Ollama and LangChain

In this video I go through setting up a basic fully local RAG system with Ollama 2 and the new Gemma 2 model.

Interested in building LLM Agents? Fill out the form below

👨‍💻Github:

⏱️Time Stamps:
00:00 Intro
00:09 Ollama: Gemma 2 Model
01:35 Demo: RAG running locally

[relaniumz]

Thank you for the video. Vote for the next video - Fully Local Multimodal RAG

[toadlguy]

Just wanted to give a big thumbs up to this, although I haven't yet watched the whole thing 😀. There are so many interesting things you can do with Local RAG and LangChain is very straight forward. I did something similar with Ollama's Llama3 model. Very interested in trying new Llama models that should be available soon.

[mrnakomoto]

useful videos. keep on uploading and make aussies proud of you

[aa-xnhc]

memory, LangChain agents, streaming UI next, please. Thanks for the very useful video!

[hrk]

Amazing tutorial, exactly what I was looking for! Running it with a few text documents, the results are great. Do you have any recommendations for making the QA faster? A different model or libraries?

[flat-line]

LangChain and lamaIndex is really is boilerplate imho, they create more problems then they solve with their over abstraction, can you show vanilla example of how to do rag ?

[henkhbit]

Thanks for showing gemma2 and ollama. Would be nice to see with mesop. Maybe in combination with langsmith for debugging?

[emmanuelauffray]

Great video again. Question though. I can see you are more in favour of langchain but what’s your thoughts on autogen and Teachable agents to do something similar? And in general I suppose your thoughts on autogen and its agentic model?

[supercurioTube]

Hey Sam! For now Gemma2 is still broken in Ollama, which doesn't include yet the latest llama.cpp fixes required.
It's about the tokenizer: <start_of_turn> and <end_of_turn> are interpreted as text instead of special tokens, and of course things don't really work as expected as a result.

I believe it'll be fixed in the next Ollama update tho - very soon. But it's too early for Gemma 2 evaluations using Ollama at the moment, like many are making on their own or publishing in videos.

[matthewpublikum]

Is it possible to run the embedding model on cpu and the localllm on gpu?

[SwapperTheFirst]

i see that it is working quite fast on Mac Mini.
But what are RAM requirements for model and chroma? Does it require GPU for acceptable performance?

You've mentioned that choice of embedder is important. As I understand the same vector dimensionality is not required, since you use embeddings only during embedding process and vector search. But what about "semantic" compatibility between embedder and LLM? I can imagine that embedder could map semantic meaning in its vector space differently from Gemma or LLama. Is it even possible to compare to ensure that you use the best possible embedder for some model?

[yazanrisheh]

Hey sam can you explain why does your prompt template always seem in a different structure? By that I mean in this case you wrote at the start <bos><start_of_turn>user\n then towrads the end you wrote <end_of_turn>. Does each llm have its own way of writing its own prompt template? If so, what & where do you refer to when you want to do prompt engineering for an llm ur using?

[PestOnYT]

I find chroma is not very suitable for local RAG. It sends back telemetry data to their devs. One needs to set anonymized_telemetry=False to keep it quiet. Also, running ollama with some of the tools mentioned behind a firewall/proxy can be a challenge.

[bluelegend]

Hello, i'm french, sorry for translate, really good job, i have a question, how do you add pfd to txt on the top of your code ? **/*.txt, *.pdf or anything ? thank you

[themaxgo]

how does this compare to ms' recently opensourced graphrag? btw there are graphrag w/ ollama implementation tutorials (2 diff versions to do it, 1 is a "hack" / req graphrag python lib change to make it work w/ ollama, other one req lm studio)... with 2 types of querying: "global", which works fine, always; "local", which often / usually fails (w/ various error msgs / for various reasons)

[ShravanKumar]

What are the system requirements?
Do we need a GPU with certain size of VRAM?

[HmzaY]

it doesn't work very well, but it is informative.

[matthewchung]

Can you share the index code? I do not see it in Github

[kungmo]

Thanks for sharing your experience.
I want to run this model on my computer. So I wrote Modelfile like below:

FROM gemma-2-9b-it-Q6_K_L.gguf

TEMPLATE """
<start_of_turn>user:
{{prompt}}<end_of_turn>
<start_of_turn>model:
"""

PARAMETER stop <end_of_turn>

And I create model to ollama, so I ran this command

ollama create ollama create gemma-2-9b-it-Q6_K_L -f

And I want to run this model, so I ran this command

ollama run gemma-2-9b-it-Q6_K_L:latest

Finally, I got an error message....

Error: llama runner process has terminated: signal: aborted (core dumped)

How could you run this model on ollama?
Thank you.