Better RAG: Hybrid Search in Chat with Documents | BM25 and Ensemble

preview_player
Показать описание
Learn Advanced RAG concepts to talk your chat with documents to the next level with Hybrid Search. We will look at BM25 algorithm along with ensemble retriever. The implementation will be in langchain.

LINKS:

TIMESTAMPS:
[00:00] Introduction to Advanced RAG Pipelines
[00:11] Understanding the Basics of RAG Pipelines
[01:49] Improving RAG Pipelines with Hybrid Search
[02:55] Code Example: Implementing Hybrid Search
[05:08] Loading and Processing the PDF File
[06:24] Creating Embeddings and Vector Store
[08:46] Setting Up the Retrievers
[12:52] Running the Model and Analyzing the Output

All Interesting Videos:

  1. Prompt Engineering
  2. prompt engineering
  3. Prompt Engineer
  4. LLMs
  5. AI
  6. artificial Intelligence
Рекомендации по теме
Better RAG: Hybrid 0:16:08

Better RAG: Hybrid Search in Chat with Documents | BM25 and Ensemble

Hybrid Search RAG 0:42:35

Hybrid Search RAG With Langchain And Pinecone Vector DB

Understanding Reciprocal Rank 0:08:16

Understanding Reciprocal Rank Fusion in Hybrid Search [Advanced RAG]

Advanced RAG 03 0:06:47

Advanced RAG 03 - Hybrid Search BM25 & Ensembles

Don't naive RAG 0:42:15

Don't naive RAG do hybrid search instead (Pinecone Weaviate or pgvector + full text search &...

Advacne RAG 01 1:01:10

Advacne RAG 01 - Powerful RAG Using Hybrid Search(Keyword+vVector search) | Ensemble Retrieval

Deep Dive into 0:25:02

Deep Dive into Hybrid search techniques in RAG: BM25, MMR, Reciprocal Ranking, Metadata filtering

Is Your RAG 0:01:23

Is Your RAG System Underperforming? 😫 Boost Results with Hybrid Search & Semantic Reranking NOW!...

GenAI 101: Getting 0:50:40

GenAI 101: Getting Started with a Vector Database

RAG But Better: 0:23:43

RAG But Better: Rerankers with Cohere AI

Hybrid Search for 0:06:30

Hybrid Search for RAG in DuckDB (Reciprocal Rank Fusion)

LangChain - Advanced 0:24:57

LangChain - Advanced RAG Techniques for better Retrieval Performance

HybridRAG: Ultimate RAG 0:09:33

HybridRAG: Ultimate RAG Engine - Knowledge Graphs + Vector Retrieval! Better Than GraphRAG!

Why you should 0:02:35

Why you should consider use Hybrid Search in your RAG pipeline

Data Science Interview 0:08:08

Data Science Interview Question : Hybrid Search in RAG

Optimize RAG with 0:58:06

Optimize RAG with hybrid search and reranking

Advacne RAG 02 0:48:16

Advacne RAG 02 - Hybrid Search (Keyword + Vector ) & Reranking With Cohere API | Ensemble Retri...

Advance RAG chatbot 0:19:25

Advance RAG chatbot with Graph | Hybrid search with Graph RAG | RAG with Neo4j and Groq API

Building Production-Ready RAG 0:18:35

Building Production-Ready RAG Applications: Jerry Liu

Vector , Hybrid, 0:11:12

Vector , Hybrid, Semantic search with Azure AI Search in 11 minutes

RAG With Azure 0:14:56

RAG With Azure AI Search using Vector/ Hybrid / Exhaustive KNN / Semantic Reranker in 15 minutes

Powerful RAG Using 0:42:14

Powerful RAG Using Hybrid Search | iNeuron

Pinecone's New *Hybrid* 0:20:29

Pinecone's New *Hybrid* Search - the future of search?

'I want Llama3 0:24:02

'I want Llama3 to perform 10x with my private knowledge' - Local Agentic RAG w/ llama3

Комментарии
Автор

Hey, These videos are really helpful. What do you think about scalability? When the document size increases from few to 1000s, the performance of semantic search decreases. Also have you tried qdrant? It worked better than chroma for me.

lakshay
Автор

How do you handle multiple documents that are unrelated to find the answer for the user ?

mrchongnoi
Автор

Great - while you can persist the chromadb, is there a way to persist der bm25retriever? or do you have to chunk always again when starting the application?

saqqara
Автор

Very nice idea with this 'code display window' in your video:
now the code is much easier to read, and much easier to follow step by step. Thanks.

poloceccati
Автор

I get KeyError 0 when I run this

# Vector store with the selected embedding model
vectorstore = Chroma.from_documents(chunks, embeddings)

What am I doing wrong? I added my HF token with read the first time and then with write too...

I would appreciate the help.

Thanks for the video, though. Its amazing.

clinton
Автор

00:01 Introduction to Advanced RAG series
02:06 Hybrid search combines semantic and keyword-based search
04:07 Setting up the necessary components for hybrid search in chat with documents
06:13 Creating and using API token in Google Colab
08:19 Creating Vector store and retrievers for hybrid search in chat with documents
10:16 Using different retrievers for different types of documents
12:11 Creating a prompt chat template for the model.
14:12 Comparison of Orca and CH GPT

quickcinemarecap
Автор

Excellent video I’ve been needing this. Very slick way to combine the responses from semantic and keyword search.

TomanswerAi
Автор

@engineerprompt - Could you convert Notebook with LlamaIndex if you don't mind?

JanghyunBaek
Автор

Fantastic Video and very timely. Thanks for the advice. I have made some massive progress because of it.

paulmiller
Автор

Hello! First of all, thank you very much for the video! Secondly, at minute 10:20 you mention that you are going to create a new video about obtaining the metadata of the chunks. Do you have that video? Again, thank you very much for the material.

SRV
Автор

It's great that the example code uses free LLM inference like Hugging Face (or OpenRouter)!

attilavass
Автор

This video is really helpful to me!Thanks a lot!

MikewasG
Автор

I'm using RAG for a coding model. can anyone suggest a good retriever for this task?. Thanks in advance!

chrismathew
Автор

Amazing video! How can you use this in a conversational chat engine? I have built conversational pipelines that use RAG, however how would I do this here while having different retrievers?

zYokiS
Автор

The background is little distracting, its better to avoid the flashy one, i couldn't concentrate on your lecture. Please. Thank you.

vamshi
Автор

i dont know what RAG to implement . is there benchmarks out there for the best solution? My use case will be 100s of LONG documents even textbooks.

karanv
Автор

Thanks! I have 500k documents. I want to compute the keyword retriever once and call it the same way I have external index for dense DB vector. Is there a way?

Tofipie
Автор

hello! thanks for the video. I was wondering if we can use it on csv files instead of PDF? How would that affect the architecture?

deixis
Автор

Thank you for sharing the guide. One question, how to make the response longer, I have tried to change the max_length parameter, as you suggested in the video, but the response is always some ~ 300 characters long.

rafaf
Автор

Hi, I have a question, hope you reply. If we want to give it a PDF with bunch of video transcripts and ask it to formulate a creative article based on the info given, can it actually do the tasks like that? Or is it just useful for finding relevant information from the source files?

PallaviChauhan