The Hidden Cost of Embeddings in RAG and how to Fix it

Embeddings are crucial for a production-ready RAG system but often get overlooked. I cover the costs, storage considerations, and ways to reduce storage requirements using techniques like dimensionality reduction and quantization. Learn how these methods can improve speed and save costs without compromising too much on performance.

LINKS:

💻 RAG Beyond Basics Course:

Let's Connect:

Signup for Newsletter, localgpt:

00:00 Introduction to Embeddings in RAG Systems
00:47 Understanding Embedding Costs
01:17 Storage Costs and Considerations
03:32 Reducing Storage Needs
03:41 Dimensionality Reduction Techniques
04:24 Matrosha Representation Learning
05:14 Precision Reduction Techniques
06:28 Quantization Study by Hugging Face
10:07 Implementing Quantization in Your Pipelines
12:56 Using Open Source Vector Stores
15:01 Conclusion and Final Thoughts

All Interesting Videos:

[greendsnow]

Use Embedding-3-small + Qdrant Quantization for saving storage costs.

[BadBite]

Pretty good! very useful as I never thought about the long term wallet bleeding

[uwegenosdude]

Thank you very much ! This is very good to know if our app is becoming bigger.

[aibeginnertutorials]

Brilliant and extremely useful and relevant information as usual. Thanks!

[MeinDeutschkurs]

That’s great! Yes, please create a video with a useful example. I‘d appreciate it! 🎉🎉

[unclecode]

Very interesting and important points you raised. I’ve seen startups completely unaware of this and, as a result, they're doomed. Many don’t even use features like OpenAI’s dimension reduction. This binary and quantization has been around since March and is incredibly powerful. Now, with Gemini's support for PDF and long context windows, freeing up to a billion tokens in a day, it raises questions about when to use embedding and RAG, and when not to. When necessary, combining this with a long context window seems like the perfect solution. I suggest you create a video showing how to use this with Gemini to fetch and cache context, which will deliver the best balance of performance and cost.

[sashirestela]

Thanks for your very useful information.

[messam]

Thank you, waiting for real tutorial for production RAG app

[jirikosek]

Just a question, I see you are mentioning e.g. the AWS X2gd EC2 instance. So if I understand correctly you want to keep all the vectors in memory. Isn't it better to just use a storage solution for this instead if the database is massive? E.g. Amazon OpenSearch Service. Storage should be cheap...

[hsin-yusu]

Yes, this is exactly what I'm looking for

[harshilpatel]

Please make a video on hybrid search using the BM25 algorithm.

[abdulrehmanbaber]

Very helpful
One question, can you explain the difference between this word quantization used with embedding model (here) and use of quantization when doing inference or fine-tuning!?

[jayco]

pinecone storage is about $0.334 per gb per month but also has other fees for read and write.

[mattshelley]

Using QDrant on our servers, RAM will be our largest expense to maintain the database as it grows.