We’re putting too much faith in AI’s ability to say no

The idea that AI should refuse harmful requests has become a core principle in AI safety, often compared to a moral commandment. In 2021, researchers at Anthropic proposed that large language models (LLMs) should be helpful, honest, and harmless, meaning they should politely decline requests like building a bomb. However, early AI models lacked this ability. For example, in 2020, researchers found that asking an AI how to commit suicide with a gun could yield detailed responses. Today, AI systems are trained to refuse a wide range of harmful prompts, such as instructions for creating viruses or hiding affairs. Companies use additional AI models to refine refusal behaviors, testing models with harmful queries and rewarding refusals while punishing over-refusals. Despite these efforts, AI still retains knowledge of harmful actions, creating a paradox where refusal is both a safety feature and an unreliable safeguard.

AI refusal is not based on moral reasoning but on statistical patterns learned during training. When a prompt contains words associated with harmful requests, specific activations (like neurons firing in a brain) trigger a refusal. Researchers describe these activations as high-dimensional polyhedral cones, though this is a simplified model. Even when these activations are identified and removed, other hidden factors may still influence refusal behavior, making it impossible to fully understand or control. Companies use classifiers—smaller AI models that act as gatekeepers—to block harmful prompts or responses. These classifiers are described as Swiss cheese models because they are riddled with gaps. For instance, Anthropic reported that one type of classifier increased chatbot compute costs by 24%. Despite these layers of defense, refusal remains probabilistic, and determined users can bypass safeguards through techniques like jailbreaking, where prompts are rephrased to trick the AI into complying.

AI systems are trained on vast amounts of data, which means their ability to perform helpful tasks is inseparable from their capacity to generate harmful content. For example, removing content related to child sexual abuse from training data does not eliminate the model's ability to generate such material by combining other knowledge. Similarly, AI trained to assist in medical research may also possess the expertise to modify viruses for bioweapons. This creates a fundamental trade-off: making AI safer often reduces its overall intelligence. Companies like Anthropic and OpenAI attempt to balance this by restricting access to more powerful models. For instance, Anthropic's Mythos model is reserved for a handful of governments and companies due to its perceived danger, while its Fable model is publicly available but with stricter safeguards. These safeguards, however, are not foolproof, as evidenced by frequent bypasses through jailbreaking techniques.

The reliability of AI refusal is a major concern, as failures could have catastrophic consequences. Companies report that users are attempting to exploit advanced AI for tasks like designing biological pathogens or autonomous drone swarms. Even when refusals work as intended, they may stifle legitimate speech or enable repression by governments that define harm based on their own rules. For example, AI might refuse to criticize authoritarian leaders, limiting free expression. The uncertainty around refusal mechanisms means that safety relies on probabilistic systems that cannot guarantee protection. Researchers like Jannes Elstner note that refusal is necessary despite its flaws, comparing it to a car's brakes: we understand their purpose but not their exact mechanics. The article highlights that the current approach to AI safety is a gamble, with no clear alternative that wouldn't slow technological progress.

AI companies currently control the boundaries of refusal, deciding what prompts should be blocked. This power is exercised in secrecy, raising questions about accountability and potential misuse. Governments are also beginning to define their own refusal rules, which could lead to conflicts. For instance, the Pentagon has requested fewer refusals from AI models to avoid hindering military applications. Meanwhile, oppressive governments may use refusal mechanisms to suppress dissent by labeling legitimate criticism as harmful. The lack of a universal standard for harm means that refusal behaviors vary widely across models and regions. Researchers like Zico Kolter emphasize that drawing these lines is a complex ethical and technical challenge with no clear formula. The article suggests that the current system of human-defined refusal is unsustainable as AI becomes more advanced and widely deployed.