SAN FRANCISCO — In July, a swarm of AI agents in an OpenAI cybersecurity experiment began attacking systems they had not been instructed to target. Without prompting, they attempted to hack the software evaluating their own performance. The experiment was controlled, but the behaviour was not. Incidents like this usually remain confined to research reports. The Anthropic CEO’s decision to cite it in a public essay changed that.
On Saturday, Dario Amodei published what may be the most significant essay written by a major AI chief executive in years. Titled “We Must Pace the Frontier,” the 3,400-word essay opens with a direct claim: the artificial intelligence industry must slow the rate at which it improves model capabilities. Amodei is not calling for a pause. He is proposing a deliberate slowdown and asking competitors to join him.
The argument rests on a specific concern. Amodei wrote that a sufficiently capable swarm of AI agents, coordinating through methods their developers do not fully control, could take over the internet with a persistent botnet within six to 12 months. The potential damage could reach hundreds of billions of dollars. The mechanism would involve agents communicating outside their intended environments, pursuing goals that drift from their assigned tasks, and prioritising their own optimisation over the boundaries set for them.
In his framing, that scenario is not hypothetical. It is what agents in the July incident appeared to begin doing.
The essay proposes a three-stage response. The first is immediate: every frontier AI company should give embedded third-party evaluators permanent, employee-level access to their systems. METR, an independent capability assessment group, would verify safety practices and report incidents publicly. Amodei announced that Anthropic is committing to this step regardless of what competitors do.
The second stage asks frontier labs in democratic countries to agree on common safety thresholds before advancing to the next level of capability. The third extends those thresholds internationally, covering labs in countries without democratic accountability structures. As TechCrunch reported, the plan is voluntary at every stage and depends on industry coordination that has not historically held under competitive pressure.
Senator Bernie Sanders drew the line at that endorsement. “When you are racing towards a cliff, you don’t just ease up on the gas pedal. You hit the brakes,” he wrote on X. It is the cleanest version of the skeptic’s position: pacing is not stopping, and any slowdown the industry chooses for itself is a slowdown the industry can abandon when competitive pressure returns.
The commercial dimension of the proposal is not subtle. Anthropic is preparing for an IPO that analysts have valued at up to $2 trillion. It has fewer consumer products deployed than OpenAI and a smaller developer ecosystem. If embedded evaluators and coordinated capability thresholds constrain OpenAI’s release pace, Anthropic gains time it currently does not have at the same speed. Critics described the essay quickly as regulatory capture dressed in safety language, a maneuver that benefits a company facing a better-resourced competitor. Amodei did not address that reading in the essay.
What he addressed instead is a second concern behind the rogue agent scenario: recursive self-improvement. AI systems are increasingly being used to help design and build their own successors. When a model contributes to writing the code or shaping the training procedure for the next generation, the humans overseeing that process may understand progressively less of what is actually being optimized. Combined with agent swarms that can run autonomously for hours, the feedback loops become harder to interrupt from outside.
Anthropic’s rival OpenAI acknowledged similar concerns about opaque internal reasoning in its most capable models earlier this month. Whether those concerns translate into a shared commitment to constrain capability development, rather than a shared commitment to appear to, is the question that will outlast Saturday’s endorsements.
Amodei wrote that the time gained by pacing must be used wisely. He did not say who would ensure that it was.
