Postgres pgvector Extension - Vector Database with PostgreSQL / Langchain Integration

In this video, we'll look at the pgvector extension for PostgreSQL, that allows you to turn your Postgres database into a vector data-store!

pgvector adds the vector data-type and distance computation operators (L2, inner product, and cosine distance) to allow you to query for "similar" items in the vector-space.

We'll see how to set pgvector up in a Docker container, and will see how to integrate it with Langchain via the PGVector object.

We'll look at how to take a piece of text, split it into chunks, create embeddings from those chunks using OpenAI, and then store the embeddings in the Postgres vector database. We'll also see how to query the database for vectors/documents that are similar to a text prompt/query.

☕️ 𝗕𝘂𝘆 𝗺𝗲 𝗮 𝗰𝗼𝗳𝗳𝗲𝗲:
To support the channel and encourage new videos, please consider buying me a coffee here:

📌 𝗖𝗵𝗮𝗽𝘁𝗲𝗿𝘀:
00:00 Intro
00:41 Introduction to pgvector for PostgreSQL
03:23 Splitting text file into chunks with Langchain RecursiveCharacterTextSplitter
06:10 Using OpenAI to get embeddings for each chunk with OpenAIEmbeddings object
10:54 Setting up pgvector and PostgreSQL in a Docker container
16:38 Using the Langchain PGVector object to connect to PostgreSQL
21:47 Finding similar vectors to a query in pgvector
25:29 Querying pgvector with SQL to get cosine distances

𝗦𝗼𝗰𝗶𝗮𝗹 𝗠𝗲𝗱𝗶𝗮:

📚 𝗙𝘂𝗿𝘁𝗵𝗲𝗿 𝗿𝗲𝗮𝗱𝗶𝗻𝗴 𝗮𝗻𝗱 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻:

#python #langchain #datascience #postgresql

[wadejohnson]

Most excellent. I am now a monthly supporter. You deserve to be paid.

[silkogelman]

Just yesterday I thought "pgvector would be interesting to see a video about".
And then you publish this! 👏👏👏🥳
Thank you Lyle. 🙏

[tejasvinnarayan]

Extremely complex concepts published in the simplest way! I could run the whole notebook typed without errors! Thank you for the clarity!

[Andromeda_]

Thank you so much for sharing the details, Your informative YouTube videos have been incredibly helpful. Great job on putting together such valuable content! Keep up the outstanding work and continue enlightening us. We truly appreciate your contributions!

[_beryl]

If anyone has a problem with getting Docker Desktop to work (you need it on Windows for example to get docker commands work) and you get WSL errors while launching it, I recommend:
- either reinstalling your WSL/updating it
or
- (what worked for me) going into your BIOS and changing the virtualization setting to enabled (if you cannot find it, you might have it under the acronym "SVM" mode).

I had a bit of a waste of time in order to make this work, was kinda dissapointed that there was no mention of the virtualization settings, but besides that (I'm being very picky) great video! For a PGVector & Jupyter beginner I had almost no problems and everything worked for me. Thanks for the vid.

[mattiassoderberg]

Fantastic comprehensive walkthrough of how to use PGVector and Python to work with vectors for your AI stuff 😀

[shinchima]

Brilliant content. Concise, no waffle. Thank you

[sqlsql]

Really appreciate your efforts you have put in for this tutorial

[AA-xzut]

fantastic content, thank you! would be great if you could do a more in depth video on how do indexing (HNSW) with the same jupyter notebook example

[teddyperera]

clear and well structured. you have an amazing style of teaching.

[fabsync]

oh man.. it has been a while and it is still the best tutorial out there.. It will be great to see something with pgvector again with django-ninja...

[australianman]

Dude thanks for making this. I always learn something from your videos. Thank you!

[Mankind]

Straight forward explanation. Thank you

[alexandredamiao]

Thank you so much for this tutorial! Very, very high quality!

[johnallen]

powerful libs - yes its almost as if AI 'needs' a highly artistic oracle to 'shape' it's 'stance' in order to focus on the goals/need of the User/app

[LearningWorldChatGPT]

What a fantastic video.
Thank you, BugBytes !

[FatimaHABIB-jmji]

Thanks,

I am having this error when creating the "vector" extension
ERROR: Could not open extension control file "C:/Program No such file or directory

[shaunpx]

loving your videos man, thank you for clear concise explanation of these topics. Do have any videos using RAG and agents in Django? I am using Django RestAPI and have been struggling with an agent controller that work fine in the notebook but then times out in my API request with the exact same code usin Char ReAct Description?

[Kingromstar]

thanks for this, it was a great help!

[helloh]

Fantastic video! Would be interesting to see a follow up on how this might work with Django?