AI Agents That Matter with Sayash Kapoor and Benedikt Stroebl - Weaviate Podcast #104!

AI Researchers have overfit to maximizing state-of-the-art accuracy at the expense of the cost to run these AI systems! We need to account for cost during optimization. Even if a chatbot can produce an amazing answer, it isn't that valuable if it costs, say $5 per response!

I am beyond excited to present the 104th Weaviate Podcast with Sayash Kapoor and Benedikt Stroebl from Princeton Language and Intelligence! Sayash and Benedikt are co-first authors of "AI Agents That Matter"! This is one of my favorite papers I've studied recently which introduces Pareto Optimal optimization to DSPy and really tames the chaos of Agent benchmarking!

This was such a fun conversation! I am beyond grateful to have met them both and to feature their research on the Weaviate Podcast! I hope you find it interesting and useful!

Chapters

0:00 Welcome Sayash and Benedikt!
1:04 Inspiration for AI Agents That Matter
4:28 Pareto Optimal Agent Optimization
21:40 Tool Use Considerations
24:54 Generative Feedback Loops
27:15 RAG versus Long Context LLM Benchmarking
30:05 Problems with Agent Benchmarks
38:45 Measuring Intelligence
44:05 Agent Architectures
49:10 Human Feedback in Agent Optimization
53:55 What directions for the future of AI excite you the most?

Links:

Please subscribe to the channel to see more episodes of the Weaviate Podcast!

[connor-shorten]

Thank you so much for joining the podcast Benedikt and Sayash! Learned so much from our chat and really excited about where this research is heading!