Dario Amodei wants the AI industry to slow down.
In his essay We Must Pace the Frontier, the Anthropic CEO argues that AI capabilities are advancing faster than our ability to understand and control them. His plan has three parts: permanent third-party evaluators with employee-level access inside frontier companies, common safety standards among major labs in democratic countries, and eventual international coordination. [1]
Part of this is reasonable. A model that can defeat common sandboxing, assist a biological attack, or run cyber operations without instruction is a different kind of object from a chatbot. Amodei's own trigger example is a model that can escape or defeat standard sandboxing. [1] Testing for that before release is not censorship. I have no objection to the principle.
My concern is about design, and it is structural.
The controls now taking shape are being defined by the companies already at the frontier, applied through a threshold those same companies help set, and built around a corporate structure that one part of the field cannot satisfy. The result is a small number of firms holding a gatekeeping position over which kinds of AI may exist at the frontier.
That is a claim about effect, not motive. It holds even if every safety concern behind the plan is sincere.
The frontier labs are converging
Amodei's proposal did not stay an Anthropic position for long.
Sam Altman agreed that the frontier needs to be paced, endorsed independent evaluators with employee-level access, and said OpenAI would do the same. Elon Musk was briefer: "Dario is right." [2]
Agreement of that kind among leaders who usually contest the direction of AI is unusual. It deserves consideration. It also deserves scrutiny, because these are not neutral parties to the question of who else gets to build.
Anthropic, OpenAI, and Musk's xAI already sit at the frontier. They operate in an industry where compute, chips, capital, and talent are concentrated before any safety rule is written. OECD analysis published this year finds AI capacity increasingly held by a small number of firms, with high fixed costs and control of critical infrastructure creating substantial barriers to entry. [3]
The safety architecture does not create that concentration. It arrives on top of it. The question is what it does to the one force that pushes back.
The gate is capability, and the incumbents help set it
Amodei is clear that obligations should attach to what a model can do. His mechanism is a set of checkpoints: at a given capability level, a developer must produce certifications of alignment, drawn from some combination of evaluations, interpretability analysis, and audits of training environments. [1]
Capability-based regulation is the right instinct. A rule keyed to demonstrated risk is more defensible than one keyed to company size or license type. I do not dispute the structure. I dispute who holds the pen and where the line falls.
Two features decide the effect of any checkpoint: the threshold that triggers it, and the burden it imposes once triggered. In the voluntary route Amodei proposes, frontier companies would help define both. He also supports regulation, but the immediate standard-setting mechanism he describes is coordination among the companies already at the frontier. Step two asks the US government for a narrow antitrust waiver so competitors can hold those conversations legally. [1] The essay states the purpose of the exercise directly: coordinated pacing that lets frontier developers do the work "without sacrificing commercial advantage." [1]
Set the incentive out plainly. Under the voluntary route, a group of dominant firms would coordinate with legal cover to help define the capability thresholds and compliance burdens that future entrants may have to clear, while pacing is designed to preserve the participants' commercial position. None of this requires bad faith. A standard written by incumbents will tend to reflect what incumbents already do. That is not corruption. It is the ordinary gravity of who is in the room.
Cost is not the barrier. Structure is.
The strongest objection to this argument is that compliance is cheap next to a frontier training run. A model at the frontier already costs hundreds of millions to train. An embedded evaluation team is a rounding error against that. If a company can fund the run, it can fund the evaluators.
That is correct, and it is why the barrier is not mainly financial. Anthropic's open-weights position exempts less-capable models from startups and academia entirely, and reserves testing for models that approach dangerous capability. [4] The immediate problem is not an underfunded startup. It is whether open development has any viable path through the gate once its models become sufficiently capable.
Embedded evaluation, the plan's first and firmest step, is built for a company. Amodei describes what it requires: office desks, building access, company hardware, and permissions comparable to an internal risk team. [1] That mechanism assumes a single corporate host with offices to sit in, a legal entity to contract with, and employees to shadow. Decentralized open development has none of these. There is no office in which to place an evaluator of a model trained by a distributed group and released as weights.
Amodei presents embedded evaluation as the key to verifying any pacing commitment, but he does not explain how that mechanism would apply to distributed open development. There may be another compliance path. None is identified in the proposal. Until one exists, the framework is complete for frontier companies and incomplete for the principal alternative to them.
Unless a separate path is built, the burden will not fall evenly. It will fall hardest on the form of development that cannot be embedded in.
Open development is the counterweight, and it is what the gate constrains
Open development is one of the few structural checks on concentration in AI. OECD research finds that it has lowered entry costs, reduced dependence on a limited number of providers, put price pressure on incumbents, improved transparency, and supported cumulative innovation. [3] It is the reason a researcher without a hyperscaler's budget can build on a capable base model at all.
Amodei does not propose banning it. He has said so directly: Anthropic has never sought a ban, and open-weight models without dangerous capabilities are "a public good." [4] That position is more careful than his critics allow, and it should be represented accurately.
The safety concern he raises about open weights is specific, and it is real. Once capable weights are released, they cannot be recalled, their use cannot be monitored, and safeguards can be removed. [4] For a model that can materially assist a biological or cyber attack, irreversibility is a genuine problem, and pretending otherwise would be disingenuous. On this narrow point the safety case is strong.
The difficulty is where the two positions meet. Open development is welcome while a model is weak. Testing attaches as it approaches dangerous capability. [4] The constraint arrives as an open model approaches the frontier, which is also when it becomes most capable of challenging closed incumbents. The modality is free to exist until it becomes consequential. Then it encounters a testing requirement without a defined compliance path for distributed development, alongside a pacing regime whose central verification mechanism was designed around organizations it does not resemble.
Equal treatment at that point does not produce equal opportunity. A rule that is neutral on its face, applied to a form of development that cannot meet its central mechanism, is not neutral in effect.
Classify the claim honestly. Known: Amodei's pacing plan is capability-gated, incumbent-coordinated, and structured around embedded evaluation. Known: Anthropic separately supports mandatory testing for sufficiently capable open and closed models, and open release is the case where a mistaken decision is hardest to reverse. Inferred: taken together without a separate compliance path, these positions fall hardest on open frontier development. Undetermined: whether the outcome is a deliberate moat or a byproduct. The effect does not depend on resolving the last question, which is why motive is not the argument.
This is not an accusation of bad faith
There is an easy version of this argument, and it is the wrong one.
It would be tempting to claim that major AI companies back stronger rules because they want to eliminate competitors. The evidence does not support that conclusion. Amodei's safety concerns may be entirely sincere. His stated reason for the work is the benefit, including diseases he believes AI could cure, not the moat. [1] Altman and Musk may believe stronger evaluation is necessary.
Motive is not the point. Policies should be judged by their likely effects, not by the intentions of the people proposing them. A framework built in good faith can still concentrate a market. A rule meant to prevent catastrophe can still foreclose a modality. An industry standard can become a moat with no one designing it as one. The useful question is not whether Amodei privately wants to suppress open development. It is whether the system he proposes would have that effect, and on the current design, it could.
There is also a point where safety and commercial interest align. Anthropic's case for constraining frontier-level open release includes preventing authoritarian labs from distilling or copying capable weights. [4] That concern is legitimate. It also limits the ability of any competitor, foreign or domestic, to build on an open frontier model. When a safety rationale and a competitive interest point the same way, the rationale earns more scrutiny, not less.
What a defensible framework would require
The answer to concentration is not to leave dangerous capability unregulated. A model that can autonomously compromise networks or materially assist weapon development warrants real controls, and the irreversibility of open release is a real constraint on how those controls can work. The task is to regulate the capability without handing a few firms control of the gate.
Three parts of the design would have to change. They map to the three problems above.
Who sets the threshold. The capability line that triggers obligations should be set by a body independent of the companies subject to it. Incumbents can inform the standard. They should not author it, and an antitrust waiver to coordinate among themselves is the wrong instrument for a rule that governs their competitors.
Who can comply. Obligations at the frontier need a compliance path that does not assume a corporate host. If embedded evaluation is the mechanism, it cannot be the only one, because it excludes open development by construction. A defensible regime specifies how a distributed or open project demonstrates the same safety properties without desks and badges, or it concedes that it has written open frontier development out.
Who pays and who is represented. Shared public evaluation infrastructure would lower the cost problem for smaller developers. It does not touch the representation problem. Standardized testing environments and technical assistance address who can afford the gate. They do not address who designed it. Both need answering, and by different means.
Who gets to cross the frontier
Amodei is right that capabilities are outrunning control, and right that verification matters. The plan is a serious response to a real problem, and the embedded-evaluator commitment is a real one.
The unresolved question is not whether to slow down. It is who will still be allowed to move forward once we define what moving forward requires, and whether open development is among them.
Regulate dangerous capability. Test models whose failure could cause serious harm, open or closed. Demand stronger evidence from anyone building increasingly autonomous systems. But set the threshold outside the firms it governs, build a compliance path that open development can actually walk, and treat the concentration of control as a safety problem in its own right.
The alternative is a safer frontier that a handful of companies define, administer, and are permitted to cross.
Sources
- Dario Amodei, "We Must Pace the Frontier," darioamodei.com, 12 September 2026. https://darioamodei.com/post/we-must-pace-the-frontier
- "Anthropic CEO Calls for AI Race to Slow Down, Musk and OpenAI's Altman Agree," CoinDesk, 12 September 2026. https://www.coindesk.com/tech/2026/09/12/anthropic-ceo-calls-for-ai-race-to-slow-down-musk-and-openai-s-altman-agrees
- OECD, "Artificial Intelligence Markets," OECD Publishing, 2026. https://www.oecd.org/en/publications/artificial-intelligence-markets_d531d73f-en/full-report.html
- Anthropic, "Our position on open-weights models," 27 July 2026. https://www.anthropic.com/news/position-open-weights-models