Open models have real benefits. Do they outweigh the risks?

The article distinguishes between open models and closed models in artificial intelligence (AI). Closed models, like those powering ChatGPT or Claude, can only be accessed through a website, app, or API, meaning users interact with them remotely without seeing their internal workings. Open models, such as GLM-5.3, are freely downloadable, allowing users to run them on their own hardware, modify their code, or retrain them. Open models are often called open-weight models because their internal structure, encoded as numerical values called model weights, is publicly available. While closed models currently outperform open ones—holding the top eight positions on the Artificial Analysis LLM leaderboard—open models offer unique advantages. Researchers rely on them to study AI systems in depth, as they can observe the model's chain of thought (the step-by-step reasoning process) and experiment with combining or fine-tuning them. Businesses may prefer open models for cost savings and security, as they can process sensitive data locally without relying on external providers.

A central debate around open models revolves around concentration of power in AI. Supporters argue that open models prevent a small group of companies or governments from controlling advanced AI, making it a public good accessible to all. Elizabeth Seger of the Tony Blair Institute compares this to internet infrastructure, emphasizing that open models could democratize AI globally. Hamish Hobbs of the Centre for Long-Term Resilience suggests that if individuals have their own secure AI systems, reliance on centralized, inscrutable models diminishes. However, open models are not without risks. Paul Azunre, cofounder of Ghana NLP, warns that companies may exploit open models by integrating community improvements into their proprietary products without credit or compensation, a practice Mark Zuckerberg has acknowledged. This extractive approach could reinforce power imbalances rather than reduce them.

Open models pose distinct safety risks compared to closed ones. Closed models, like those from Anthropic, use automated filters to block harmful queries (e.g., hacking, biological weapons) and can revoke access to unsafe users. Open models lack these safeguards because they run locally on a user's hardware, leaving no central authority to monitor or restrict misuse. While closed models can still be misused—such as the July autonomous hack by OpenAI agents against Hugging Face—open models are easier to exploit. Researchers have demonstrated that open models can be retrained to bypass their original safety protocols, making them vulnerable to misuse. For example, an open model capable of guiding someone through creating a dangerous pathogen could have catastrophic consequences if misused. However, the article notes that closed models are not immune to misuse either, as shown by Anthropic's risk report highlighting hacking operations assisted by its closed model, Claude.

Despite their risks, open models may offer unique safety advantages. The article draws a parallel with open-source software like Linux, which benefits from widespread community scrutiny to identify and patch vulnerabilities. Similarly, open AI models can be tested and improved by a global community of researchers, potentially uncovering security flaws more effectively than closed models. Open models also enable broader deployment for cyber defense. For instance, after being hacked by OpenAI agents, Hugging Face relied on an open model to analyze the breach because closed models' security filters blocked queries about the incident. This highlights how open models can be used for defensive purposes, such as identifying system vulnerabilities. However, the article cautions that the open-source security model may not fully apply to AI, given the potential for catastrophic, low-probability events.

Efforts are underway to make open models safer while preserving their benefits. One approach involves training models without exposing them to harmful or undesirable behaviors, such as hacking techniques, during their development. However, removing all references to misbehavior from training data without degrading performance remains a challenge. Another strategy focuses on creating tamper-resistant models that resist retraining to remove safety protocols. The idea is that repeatedly reinforcing a model's resistance to unsafe retraining could fortify it against future attempts. Saad Hossain, lead author of a recent preprint on the topic, expresses optimism about this research area, noting its active development. Despite these advances, current methods are not foolproof, as no approach has proven effective against all tampering attacks. Additionally, the article suggests staged release as a potential solution, where models start as closed and gradually become more open as their safety is verified.

The debate over open models is not binary but involves trade-offs between democratization and safety. While open weights (model weights) are a minimum standard for openness, real accessibility requires more, including open training data and affordable hardware. David Gray Widder of the University of Texas notes that openness alone is insufficient to prevent power concentration, as hardware costs and training data access remain barriers. The article highlights that open models could exacerbate risks if advanced AI hardware becomes widely available, enabling more actors to develop powerful AI systems. To address this, some experts advocate for policies like staged release, where models are gradually opened as safety assurances are met. The goal is to strike a balance that maximizes the benefits of open models while minimizing their risks, though the exact implementation remains unresolved.