Decoding AI's Blind Spots: Solving Causal Reasoning

Great response from the AI community on my prompt, testing causal reasoning, where all LMMs failed.

Here now my response.

#airesearch
#aieducation
#failure

[OumarDicko-ci]

Love the video, can't wait for the next. Thank you. Building a hybrid model will be a banger video.

[MBR]

Thank you so much for your content!
So for the 7th student thing, I think I understand why LeChat worked but not Claude / ChatGPT: LeChat likely did not have the "reasoning training" (or the meta prompt with all the examples) that the more recent models have, and therefore was not "tricked".
If you have not come across this article / team, I would love to understand it more "Transformers meet Neural Algorithmic Reasoners" by the team led by Petar Veličković at Deepmind which is likely one of the most interesting teams as they do research in topology (group theory etc)

[李純心-yu]

Not sure there is the same response in difference languages for 7th student question. However, we did find GPT-4o can response correct anawer in Tranditional Chinese.

[manslaughterinc.]

At the beginning, you talk about wanting the model to give you an answer based on common language. Specifically, the word 'Recieved' in the prompt. This specific problem can be solved though a Theory of Mind + Re-ask step. Have the model ask itself, "What is the user actually thinking, " then "How can I ask this question better?". This solves a significant amount of failures caused by poor prompts, since the LLM is answering questions in language it is more familiar with. It brings the question into a vector space that is more aligned with its own knowledge. This of course does not solve problems the LLM isn't trained on. It just reduces failures on problems the LLM is trained on.

[BeOnlyChaos]

Would love to hear your thoughts on LLMs doing so badly on arc-prize.