OpenAI's New GPT 3.5 Embedding Model for Semantic Search

In this video, we'll learn how to use OpenAI's new embedding model text-embedding-ada-002.

We will learn how to use the OpenAI Embedding API to generate language embeddings and then index those embeddings in the Pinecone vector database for fast and scalable vector search.

This is a powerful and common combination for building semantic search, question-answering, threat detection, and other applications that rely on NLP and search over a large corpus of text data.

Everything will be implemented with OpenAI's new GPT 3.5 class embedding model called text-embedding-ada-002; their latest embedding model that is 10x cheaper than earlier embedding models, more performant, and capable of indexing ~10 pages into a single vector embedding.

🌲 Pinecone docs:
Colab notebook:

🎙️ Support me on Patreon:

👾 Discord:

🤖 AI Dev Studio:

🎉 Subscribe for Article and Video Updates!

00:30 Semantic search with OpenAI GPT architecture
03:43 Getting started with OpenAI embeddings in Python
04:12 Initializing connection to OpenAI API
05:49 Creating OpenAI embeddings with ada
07:24 Initializing the Pinecone vector index
09:04 Getting dataset from Hugging Face to embed and index
10:03 Populating vector index with embeddings
12:01 Semantic search querying
15:09 Deleting the environment
15:23 Final notes

[jamesbriggs]

If you see "401: API Key is invalid" when initializing the Pinecone index switch your environment from "us-west1-gcp" to "us-east1-gcp"

The reason for this is that as of 23 Jan 2022 the default environment in Pinecone changed from us-west1-gcp to us-east1-gcp, so newly initialized default projects (for new users) will be using the new default env

[ChocolateMilkCultLeader]

This one video is more useful than 99% of the ChatGPT videos created by influencers these days

[carlhlazo]

I really appreciate you taking the time to draw the diagram to explain the process. Pictures do imbed themselves in our minds and for me, it really helped me understand much better. Thank you.

[MightyMoud]

I have to say you are my go-to person when it comes to NPL stuff. I love your work man. Please don't stop.

[Legotron]

Wow. I've watched maybe 7 other videos about Embedding model and example use cases before your video... this here is by far the best. Well explained, walked away with a way better understanding of Embedding than the previous 7. Will definitely come back to your video for reference. THANK YOU!!!!

[beecee]

Searching google for this exact topic led me here and to my surprise this video was just released today!

[zevictor]

I just arrived in this channel and I can't believe how good it is! Keep going, my friend. I already can't wait to see the new videos coming 🤩

[slayermm]

embedding should give this video highest score realtive to this topic

[TheRonellCross]

The amount value bombs you dropped in this video is insane. Thanks for sharing this video.

[constantinegeist]

Thinking of adding this stuff to my experimental chatbot based on flan-t5 model. Continously store dialogs in a vector database and hopefully it will forever remember various facts you told it, which is not possible with the vanilla model because of context length.

[UserCommenter]

100% subscribed after finishing the video wow you’re tops

[dbiswas]

If I have several mailing address with Name of the person. Can it be used for matching & deduplication purpose? Any suggestion?

[kylespindler]

I would be highly interested to see the performance of this model against other open source models to see if its worth justifying the price. Perhaps a good idea for your next video. How to evaluate language models. Do some kind of comparison against other known SOTA models.

[iclick]

Great vid. Remaining questions I have:

1) How do you determine the optimal text input length/type? e.g. when splitting up text content/text data at 6:06

2) If you make an embedding db for one model, is it [ever/under what conditions is it] transferrable to another model? ada 002-> ada 003?

[viemingtan]

Hi great video! I learned interesting and cool stuff. Thank you!

[chenpaul]

Nice demo! When the search results ranking is not what we want, how can we feedback to the model to improve the search ranking results?

[mrchongnoi]

Great video. I was able to prototype a small project.

[veliea]

everytime I run input"], engine=MODEL)` i get different vector respresentation. Why? Isn't it supposed to be deterministic?

[lionardo]

Is there a self hosting solution to this indexing process? I don’t want to send data to openai to do the embedding. Can I have pinecone running locally?

[mitchellstewart]

Why would you pass multiple strings in each embedding creation this will return one output? What is the meaning of this please I am trying to understand storage practices for pinecone and this has confused the process a little bit for me.