ACL 2024 Keynote: Can LLMs Reason & Plan?

Abstract: Large Language Models (LLMs) are on track to reverse what seemed like an inexorable shift of AI from explicit to tacit knowledge tasks. Trained as they are on everything ever written on the web, LLMs exhibit “approximate omniscience”–they can provide answers to all sorts of queries, but with nary a guarantee. This could herald a new era for knowledge-based AI systems–with LLMs taking the role of (blowhard?) experts. But first, we have to stop confusing the impressive style/form of the generated knowledge for correct/factual content, and resist the temptation to ascribe reasoning, planning, self-critiquing etc. powers to approximate retrieval by these n-gram models on steroids. We have to focus instead on LLM-Modulo techniques that complement the unfettered idea generation of LLMs with careful vetting by model-based verifiers (the models underlying which themselves can be teased out from LLMs in semi-automated fashion). In this talk, I will reify this vision and attendant caveats in the context of our ongoing work on understanding the role of LLMs in planning tasks.

[littlecoder]

Interesting and really thought provoking with all the hype around llms being AGIs and eating of humanity. I think it is very important to establish what they cannot do and I'm really glad that people like you are doing it at scale

[mulderbm]

Love this talk. Not only what it cant do well but also what it does well token prediction

[vectorhacker-r]

You make me want to enroll at ASU, which I am considering for my masters in computer science

[prasadaluganti]

I like your Telugu title of your presentation!

[sudhamjayanthi]

always a telugu, doing cutting edge stuff! 🫡