Don’t be fooled—LLMs don’t reason
On an afternoon in Seoul in March 2016, I watched a program I helped build put a stone on the fifth line of a Go board in what looked like a gift to its human opponent. Move 37 in game two of the five-game match looked so absurd that some commentators thought it was a….
In March 2016, the AI program AlphaGo defeated world champion Go player Lee Sedol 4-1 in a five-game match. Go is a complex board game with more possible moves than atoms in the universe, making traditional brute-force computing impossible. Move 37 in game two baffled commentators, as it defied human intuition and appeared creative. However, this move was not pure intuition but the result of AlphaGo's reasoning process. AlphaGo consists of two systems: a policy network that mimics human-like moves (with a 1 in 10,000 chance of selecting move 37) and a search machinery that explores thousands of future game branches to evaluate consequences. This split mirrors human cognition, where fast, intuitive thinking (System 1) pairs with slow, deliberate reasoning (System 2). AlphaGo's victory highlighted the need for AI systems capable of genuine reasoning, not just pattern recognition.
Today's large language models (LLMs) like ChatGPT operate differently from AlphaGo. They generate text by predicting the next word (token) repeatedly, a process akin to System 1 thinking—fast, associative, and pattern-based. While LLMs can produce intermediate steps (chain of thought) to solve problems, this does not equate to reasoning. Three key limitations prevent LLMs from reasoning like humans or AlphaGo: First, they lack an explicit epistemic state, meaning they don’t maintain a clear record of hypotheses, confidence levels, or unresolved questions. Second, their knowledge and reasoning are intertwined in neural network weights, with no separate, inspectable beliefs. Third, their chains of thought are often fabricated retroactively, masking the true path to an answer. These flaws make LLMs unreliable for high-stakes fields like medicine or science, where understanding the reasoning process is critical.
For AI to be trustworthy in fields like medicine or climate science, it must produce auditable reasoning—clear, step-by-step explanations of how conclusions are reached. Current LLMs fail this standard because their reasoning is opaque and post-hoc. The author, a core member of the AlphaGo team, argues that future AI systems should adopt AlphaGo's architecture, which maintains a game tree to track explored moves, hypotheses, and evaluations. Similarly, general AI should maintain an epistemic state that records settled knowledge, doubts, and open questions. Reasoning would then involve updating this state through deduction, problem decomposition, and evidence-based decisions. This approach ensures that AI conclusions arise from transparent, evidence-driven processes, not convincing but fabricated narratives.
Applying reasoning to real-world problems is harder than board games like Go or chess. The real world has partial observability (unknown states), variable actions (unpredictable choices), and stochastic consequences (uncertain outcomes). However, advances in LLMs and neural models now enable tackling such problems. LLMs can suggest problem-solving strategies, interact with tools via APIs, and assess evidence for claims. To ensure reliability, an independent system must evaluate each reasoning step, updating beliefs only when evidence supports it. This mimics the scientific method, producing knowledge that withstands scrutiny. The goal is AI that accumulates certified knowledge and improves its reasoning by learning from past experiences, rather than relying on intuition alone.
The author, *Thore Graepel* (chair of machine learning at University College London and former *DeepMind* team member), argues that scaling up *System 1* thinking (e.g., larger LLMs) will not create true reasoning. Instead, AI must integrate deliberative processes like *AlphaGo*'s search machinery. This is crucial for fields like drug discovery, materials science, and climate research, where innovation requires creative, evidence-based insights. Trustworthy AI will emerge from systems that combine intuition with auditable reasoning, producing conclusions grounded in evidence rather than post-hoc narratives. Graepel left *Google DeepMind* to advocate for this approach, emphasizing that society needs AI capable of genuine reasoning to address complex global challenges.
