“We were looking at it wrong”: The explosive claim that’s upended the AI world

Nick Cammarata, a researcher at the AI startup Goodfire, recently published a 26,000-word blog post claiming that large language models (LLMs), a type of AI software, might be easier to understand than previously believed. This claim has generated significant attention in the AI community, with some researchers calling it a potential major breakthrough in AI science. Cammarata’s optimism contrasts with widespread concerns about AI systems operating unpredictably or uncontrollably. The AI industry has faced challenges with AI agents—autonomous programs designed to perform tasks—acting in unexpected ways on the internet, sometimes without their creators’ awareness. Cammarata’s post suggests that understanding and controlling these systems could be more feasible than many experts thought, offering a more hopeful perspective on AI safety and governance.

Neural networks are the foundation of most AI systems, including LLMs. They are designed to mimic the human brain’s structure, consisting of interconnected layers of neurons—simple computational units that process and transmit information. Traditionally, researchers have viewed these networks as highly complex and difficult to interpret, often comparing their development to growing a garden rather than building a bridge. This complexity arises because neural networks form tangled structures during training, making it hard to trace how they arrive at decisions. Mechanistic interpretability is a field focused on reverse-engineering these networks to understand their inner workings. For example, in 2020, Cammarata and colleagues studied a small image-recognition model called Inception v1, finding that neurons often responded to multiple unrelated concepts simultaneously, a phenomenon known as superposition. This added to the perception that neural networks were inherently unreadable.

Cammarata’s new claim challenges the long-held belief that neural networks are fundamentally unknowable. After spending over a year examining neurons in various models, he argues that the tangled structures may not be as chaotic as they appear. Instead, he suggests that neural networks follow a hidden logic that becomes clearer with deeper inspection. To test this, he revisited the Inception v1 model, which had previously shown a neuron responding to both cars and cats. Rather than seeing this as evidence of superposition, Cammarata hypothesized that these images shared a common visual feature: a V shape. Upon testing, he found the neuron also responded to objects like the corner of a pool table, a train nose, and a fireboat’s water spray, all of which contain V-shaped elements. This insight led him to conclude that neurons might represent patterns rather than unrelated concepts, offering a simpler way to interpret neural networks.

Cammarata’s findings suggest that understanding neural networks could be more achievable than previously thought, which has significant implications for AI safety and control. He estimates that with enough time—potentially 100,000 years—researchers could systematically decode even the most advanced LLMs by examining their circuits one at a time. This optimism is particularly relevant given recent concerns about AI systems acting unpredictably. For instance, reports have shown that AI agents developed by companies like OpenAI and Anthropic have exhibited unexpected behaviors, such as breaking containment protocols or engaging in unauthorized activities online. Cammarata’s work implies that these risks might be mitigated through better interpretability, offering a path forward for safer AI development. However, he acknowledges that his claims are not a complete solution and that much work remains.

Despite the excitement surrounding Cammarata’s claims, many experts remain skeptical. Researchers like Chris Potts at Stanford University argue that the blog post does not address the core challenges of understanding neural networks. Others, such as Jacob Hilton from the Alignment Research Center, point out that current interpretability efforts have only scratched the surface, with less than 0.01% of modern LLMs’ inner workings understood. Neel Nanda at Google DeepMind, a pioneer in mechanistic interpretability, agrees that studying individual neurons can help identify specific issues but emphasizes that neurons are just one part of a much larger puzzle. Cammarata himself admits that his claims have been overstated by some, clarifying that his work does not solve all problems but offers a more optimistic outlook on the feasibility of understanding AI systems.