RAG from scratch: Part 9 (Query Translation -- HyDE)

HyDE (Hypothetical Document Embeddings) is an approach to improve retrieval that generates hypothetical documents that could be used to answer the user input question. These documents, drawn from the LLMs knowledge, are embedded and used to retrieve documents from an index. The idea is that hypothetical documents may be better aligned with the indexes documents than the raw user question.

Slides:

Code:

Reference:

[ahmed_hefnawy]

the worst thing about this playlist is it's ending...
thanks for the amazing explanation, your effort is really appreciated,

wish to update it ASAP

[MrIsaacbabsky]

I love the lack of b.s. that these videos have... genius simple explanation, very pragmatic, well structured and super practical... I'm Lance huge fan!!

[sarveshshirude]

I like to become a Software Engineer like Lance Martin, the way he explains everything swiftly inspires me🙏

[freemindsupplements]

I love the creativity in this approach!

[ncde]

Thank you .. all great videos. Learned about RAG and LangChain from scratch through this playlist

[AppsWithCode]

This is really neat. Thanks for the awesome series.

[sylap]

Lance, we can't wait for the updates :(

[drizzle]

there's this part that confuses me, if you are asking a question that you have no knowledge of, say regarding private documents that are only stored in the vector db, then how would you be able to generate hypothetical document? Or in other words, if you already have the information to generate hypothetical document, why bother doing the query? doesn't this defeat the purpose? I know I must be missing something here, thanks ahead if someone can help clarify a bit.

[CostaMichailidis]

Would be nice to see the differences in distance for one technique versus another.

[rutambhagat]

I am a little confused with this technique, isn't it making the RAG app prone to hallucinations?

[elyxavier]

Dear Lance, when is the next set of videos from this series coming up (i.e., routing and query construction)? Thanks!

[manideep]

I really like the videos. I just to understand how can we evaluate the performance (quality of response) of different query translation techniques for a specific use case or domain.

[guillermogaete]

Nice video! But I think that is needed to use a diferent documents, the retrieval for back propagating docs was the same as que original question

[bingyaoli]

What scinario is the best to use HyDE?

[aidan_kang]

I can see this being problematic in domains that the model is not trained on. If it produces a hypothetical document that is way off the rails, that hypothetical document won't return good results.