Session 7: RAG Evaluation with RAGAS and How to Improve Retrieval

What you'll learn this session:
- How and why to evaluate RAG systems using best-practice open-source tooling
- RAG Assessment with RAGAS, including Context Precision, Context Recall, Answer Relevancy, and Faithfulness
- How to improve RAG system outputs using advanced retrieval

Speakers:
Dr. Greg Loughnane, Founder & CEO AI Makerspace.

Chris Alexiuk, CTO AI Makerspace.

Apply for one of our AI Engineering Courses today!

[enceladus]

incredibly informative, not like clickbait or anything like other channels. real 37mins worth of knowledge. Thank you 🙌

[lespaceman]

Great presentation guys, full of valuable knowledge 🎉

[wfmokcy]

This is really great explanation. I have one query, lets say I want to improve the performance by focusing on Faithfulness or Answer Relevance, so which RAG optimization techniques I should follow to increase Faithfulness or which techniques can improve Relevance or Precision etc.

[marnow]

Great video! How can I use RAGAS with Azure OpenAI flavour?

[supergaulig]

Good video but one question: Why did you choose to create the testset step-by-step yourself and not use the provided TestSetGenerator from Ragas? Was is not available back then?

[RaviPrakash-dzfm]

Can anyone tell me how ragas actually calculates these numbers. Like manually I get it, but what do the algorithms or functions look like? Like how does it measure faithfulness?

[farhangnorouzi]

Would you share the link to the notebook please??

[andybrown]

Thanks for the great video. When did context relevance get broken out into context precision and context recall? The RAGAs paper of 26 September 2023 still refers only to relevance and I'd find it useful to have a source to explain why it was broken into two components. Intuitively it makes sense though.

[farhangnorouzi]

Thanks for sharing. I’m looking for a github link to its repo, if possible

[cynogriffin]

Hi Chris, Very informative video, Can you please tell how can I generate test set using Azure in RAGAs.

[HosselBossel]

Chris I love your explanations and notebooks! But you shouldn't be singing while Greg is talking at 16:49

[kamalyadav]

Hi chris
I have a use case for text-to-SQL with RAG using LangChain. Is there any example or guide to evaluate the SQL result? Is the metric the same as regular text RAG? Thanks in advance

[privacytest]

Ground truth generated by GPT-4? Not even remotely useful for local RAG! In fact, ground truth presupposes you know the question, not really typical of real world user interactions.

[AdamPippert]

Why did nobody laugh at Greg’s durag joke?

[nirash]

Dude you're over 30 years old. Take the cap off if you want to be taken seriously