Open AI Embeddings in Azure Vector Database of Cognitive Search

Connect Open AI Models to your Data using the new Vector database of Azure Cognitive search for having hybrid search indexing ( based on both word embeddings and semantic search) for Retrieval Augmented Generation (RAG) or chatGPT on your own data.

Use ChatGPT on your own large data video:

Reference code used in this video in the discord channel under the reference section:

*******************
LET'S CONNECT!
*******************

✅ You can contact me at:

🔔 Subscribe for more cloud computing, data, and AI analytics videos
by clicking on the subscribe button so you don't miss anything.

#ChatGPT #AzureCognitiveSearch #wordembeddings #vectordatabase #OpenAI #AI #artificialintelligence #machinelearning #cloudcomputing #chatbot #virtualassistant #customerservice #developers #technews #innovation #Microsoft #CognitiveServices #BotService #naturalanguageprocessing #NLP #conversationalAI #videotutorial #tutorial #learningAI #AIchatbot #AIassistant #AzureAPI #AzureTools #MG #AzureMG

[shamaldesilva]

Was wondering what the azure equivalent was to vector databases like pinecone, thank you so much MG 🎉 appreciate your hard work and dedication 🥳

[kevindibb]

Excellent cog search tutorial as usual, but honestly I needed to hear the last minute and half of this video more than anything else. Thank you.

[dbiswas]

One of the best content, and the end spiritual teaching was awesome too. Thanks 🙏

[vijayakannanr]

Excellent Video.
Say, if our document repository grow in size on daily basis. Creating the embedding of the newly added documents is not a problem but the time taken to index the whole vectors again is time consuming. How can we reduce the time of indexing?

[Yanikikudon]

🎯 Key Takeaways for quick navigation:

00:06 🤷‍♂️ Struggling with deciding between using Azure Cognitive Search and creating a vector database
- Discusses the struggle of deciding between using Azure Cognitive Search for indexing words retrieval or creating a vector database with word embeddings.
- Mentions the possibility of taking a hybrid approach.
01:16 🤝 Introduction to the hybrid approach
- Discusses that Azure Cognitive Search now has the capability to store word embeddings as a vector database and index them, enabling both semantic search and word embedding based retrieval.
- Suggests using this approach for chat with your data scenarios.
03:44 🔎 Deep Dive into Vector Search
- Highlights the details and the best practices of using Vector Search.
- Explains that Vector search uses Azure Cognitive Search as a vector database to store generated word embeddings for text, images, and videos.
07:16 🌐 Multilingual Search Capability with Vector Search
- Talks about the ability to perform searches using Vector Search regardless of language, thanks to the vector representation of the context.
- Briefly mentions the use of vector search to support semantic search and word embedding based search.
08:54 💾 Creating an Index for Word Embeddings in Azure Cognitive Search
- Demonstrates a Python code showing how you can create an index for word embeddings using Azure Cognitive Search.
- Specifies required credentials and configurations to connect to Azure Open AI and Azure Cognitive search services.
14:25 🗣️ Conversation with Data using Lang Chain and Cognitive Search
- Uses LangChain for managing the conversation with the data, using Cognitive Search for retrieving the information based on word embeddings.
- Shows how questions are answered based on the closest word embeddings.
16:34 🔍 Digging into the Backend of Cognitive Search
- Looks into the backend of Cognitive Search, showing how word embeddings and data are stored and indexed in Azure Cognitive Search.
- Asserts the value and efficiency of using Vector search in the retrieval process.
18:48 🌟 Highlighting the Value of Vector Search
- Emphasizes the value of Vector search and its advantages in making retrieval of information simpler and more efficient.
- Encourages followers to use the hybrid approach leveraging semantic search and word embeddings based search.
21:06 💡 Final thoughts and Philosophical ending
- Discusses the concept of imperfection and the power of forgiveness.
- Encourages viewers to dream big, believe in themselves, and take action.

Made with HARPA AI

[learnwithengineer]

Hello I have a doubt if I am using RBAC in my azure AI search how can I Create a connection? below code we have to pass Key but if I am using RBAC how can I create Connection
acs = AzureSearch(
azure_search_endpoint=endpoint,
azure_search_key=azureaikey,
index_name=index_name,
embedding_function=embeddings.embed_query,
)

[pylanookesh]

Thank you MG, for the clear explanation.

[bramjanssen]

keep up the good work! great tutorials!

[bobetko]

What about data storage? Where is original data and data embeddings stored?

[sandeeppatidar]

Thank you so much for this wonderful video. Do we have any trade off for hybrid search? Let’s say we have 1000 of pdfs ingested in azure congnitive index and also use same index for embedding vectors and then we do a hybrid search, so seach will take more time in hybrid search compare to only vector or semantic seach ?

[michaelwindeyer]

I have a similar question @shamaldesilva9533 as to if it is similar to pinecone and also, as a first time viewer of yours, I wanted to say the spiritual message at the end was unexpected but AWESOME!

[mihaelamironescu]

How and where do i push this code to azure and make it run remotely?

[student]

Does small chunk size matters ? What happen if my question is between 2 chunk ?

[JohnLee-wvwq]

Can you point me where is the code you are using?

[bqecnto]

do you have this done in nodejs by any chance?

[AmirSharifianasanet]

Really Interesting MG, Thank you, Your accent is transforming from persian English into Indian English, maybe because your colleagues in MS are mostly Indian :) kidding boro :D.

[vikassalaria]

I am getting errror in Azure Search while connecting with cognitive search, it says resource not found, although i am grabbing correct resource as cognitive search url and key.i have models and cognitive search in same region, East US