Using pgvector + PostgreSQL in Python for vector storage and querying

I'll demo this repository that I made today:

It's a dev container with a pgvector image, plus the pgvector Python package and common packages for using Postgres from Python (psycopg2, SQLAlchemy, SQLModel, asyncpg). I'll walk through the basic demos, and finish with a demo storing ada-002 embeddings.

Links from the stream:

[ff]

Great job stepping through a pgvector setup fast!

[SiarheiKarko]

Hello Pamela, while searching for a pgvector tutorial, I came across your video recommendation. Typically, I rapidly scroll through content, seeking specific answers to my questions. However, your video proved so engaging that I watched it in its entirety, from start to finish. Thank you for your dedication and for creating such compelling content. Please continue to share your amazing videos!

[pshar]

Hi Pamel. What if we have millions of records each having vector of 1024 embeddings. How can we insert all these records efficiently in postgres. And how to keep table updated when embeddings change, i.e upsert operation on the table.

A real life example might help your viewers more. Something like a table with vector(1024) column and inserting 10 million records in it and keeping it upto date with every run of the pipeline.