Upgrade to multi-AI: Update Vector DB to AI

Vector Stores or Vector Database means currently big business. Good news: You can build your own Vector store for free. And I show you how to upgrade your Vector DB to an AI system. For free, w/ or w/o LangChain. ....... Science, not science fiction. On what components of your LLM LangChain AI system to save money and how to upgrade your LangChain Vector Store (about a dozen commercial provider) to an AI? Simple answers and integrate two interactive AI systems in your LangChain, with minimum costs.

Save money on external service providers and build your optimized, second AI right next to GPT-4, interacting with each other? Yes, but only if you know how to code. Why not start today?

A multi-AI system with two interactive AI (GPT and BERT), with and without autonomous agents.

#ai
#naturallanguageprocessing
#datascience
#gpt4

[NNRxCR]

The timing on this video is perfect. I was literally looking at options for alternatives to using openai's costly ada embeddings especially when I'm working with thousands of pages for a specific domain. I think it would also be interesting to see alternatives to popular vector databases like pinecone. Maybe a comparison of new ones that are being developed, especially open source dbs.

[hablalabiblia]

You are the most important asset Open Source deep learning has. Hope to meet one of these days.

[Davipar]

You are the best AI youtube channel out there mate. Thank you so much for you work and sharings!!

[jayhu]

I am very glad to find your channel, with your knowledge, people can learn and save money locally with this innovation.
Because otherwise it is too expensive for us to make his own business. I appreciate that your share your wisdom.
Hopefully more from this stuff. Many thanks.

[SebastianSmith]

Amazing video! Thanks for all that knowledge. Just came in the right time! :) Best wishes from the neighbour, Hungary!

[danson]

this video + future opensource replacement for gpt4 and we are there;-). Thanks again Mr!

[Esteband]

Each of your videos is a masterclass !
Thank you.

[blendercomp]

Awesome content - as always! :)
I'm really looking forward to running locally an open source LLM so that the need for OpenAI is completely eliminated!

[littlegravitas]

Hi, I have a project I'm working on with so friends (as a proof of concept in my own research in intelligent systems) and your videos are going to be invaluable! Is there any way we can reach out to you? I'd really love to ask a couple of questions, but I appreciate if you're too busy, the content you provide is so much already!

[sirrr.]

I am not a programmer but since I have started understanding LLMs I an idea that general purpose embeddings won't do anything for any domain-specific work. But I didn't know how to do this myself. Glad that you have made this video.

I have one more question that how can we train an open source LLM like gpt 4 all for domain specific storytelling like GPT 4 or even chatGPT 3.5 turbo. Please make a detailed tutorial on getting custom prompt outputs from any LLM model. And also give us a practical example of how can we prepare our own data for inputting into our own embeddings.

One last thing is how can we download a huge pile of data automatically?

I know these are too many requests but I am really taking interest in building something for my own country now which no one has built before. 😁

[toddnedd]

Thanks for the video. Unfortunately, I have not yet fully understood the complete process in detail. Maybe you can give some more depth information about that.
1. Do you use only the first column [cls] of the embedding matrix for the similarity calculation?
2. How do you score the results of the similarity calculation and rank them?
3. If you find the documents with the highest similarity (lets say top 5 documents) what is the content of the documents that you pass to gpt-4, the paragraphs with the highest similarity or some summary of the document or do you use some lang-chain approach (document map)?

[WillMcCartneyAI]

Is the core ramification of this specific workflow that a person need not use an extra vector database - essentially a customised SBERT AI would be enough?

[dtkincaid]

Very helpful video. I have to go check out your sbert videos now too. If you are not using a database of some sort to store the embedding of your documents, are you recalculating all the embeddings whenever your app starts up and putting them in memory? Would you use something like FAISS which is free to use to create a vector index in memory? Would that offer better performance than hand coding the search algorithm?

[Cloudvenus]

Another amazing video! Relying on Openai entirely is a very costly endeavor. Especially if one uses third party vector database too. Do you think we can use peft adapters for sparse retrieval and rerank? Or do we need to fine-tune all layers without freezing any parameters for this to work effectively? If we can utilize LoRA, that would be awesome, but from my understanding, there is a slight trade-off on accuracy.

[AdamBrusselback]

So, I think I understand, but I'm not quite sure and wanted to clarify. I can understand using an encoder model rather than a vector DB for things like scientific papers in your industry and all, but wouldn't you still want a vector database for canonical files you are working with for example, within your company?

Wouldn't your SBERT integrate nicely with a vector database to provide context/domain sensitive tokens for similarity search over your vector DB dataset, while also provide a secondary set of similarity tokens for "general industry knowledge" on the topic?

[MaherRagaa]

Hello @code_your_own_AI, Thank you very much for this valuable and informative content 👍. We tried Embedding for Arabic language using various models but we could not achieve acceptable accuracy in Symantec search, any recommendations? Thanks 😊

[creativeuser]

Does the embedding method create a self-attention matrix on the input sentence ?

[WillMcCartneyAI]

have you got any thoughts on preserving privacy? Will this require using an LLM other than GPT-4 for completions?

[constantinebimplis]

Great work thank you! I don't like paying for embeddings as well. You mention Huggingface embeddings which are great. My question to you is, how do hf embeddings compare to llama.cpp embeddings which have been released recently with 4096 dimensions? Would you use these? My use case is semantic search in large volumes of text. Thank you

[thezhron]

There's a new pre print talking about "Semantic Tokenizer for Enhanced Natural Language Processing" . I think it would be a golden Tokenizer, maybe? 😅