The Quiet Capture: How Pentagon Safety Testing Became AI's Real Regulatory Moat

On May 5, the Commerce Department announced that Microsoft, Google DeepMind, and xAI had signed agreements letting the U.S. government test their frontier AI models before public release. The office doing the testing is called CAISI¹ — the Center for AI Standards and Innovation, housed inside the National Institute of Standards and Technology. With the renegotiated MOUs already in place with OpenAI and Anthropic, every major U.S. frontier lab² is now routing pre-release models through a single federal office that didn't exist in its current form two years ago, has fewer than 200 staff, and operates without statutory authority, notice-and-comment rulemaking, or judicial review of its determinations.

The official framing is light-touch: voluntary information-sharing, no fees, no licensing gate, no civil penalties. CAISI Director Chris Fall called it "measurement science" in service of national security¹. Industry trade groups have applauded. And on paper, that framing is accurate. The MOUs themselves do not impose obligations Congress hasn't authorized, because the MOUs do not impose obligations at all. Labs hand over models with safeguards stripped, government evaluators probe them in classified environments, and everyone goes home.

I think that framing is also misleading, and I want to explain why — carefully, because the most common version of the "regulatory capture" critique overstates the case in ways that are easy to refute. The interesting argument is narrower and stronger.

Start with what the MOUs are not. They are not a licensing regime. Nothing in the May 5 agreements bars Meta or Mistral or AI2 from releasing models, and nothing requires non-signatories to pay a fee or seek pre-clearance. The UK's equivalent body has open-sourced its evaluation framework³ under an MIT license, and CAISI has publicly evaluated open-weights models, including Chinese ones. Government evaluation capacity, in the abstract, is a public good. Someone needs to know whether a frontier model can meaningfully uplift a bioweapons program or a cyberattack chain, and "trust the lab's internal red team" is not an answer that scales. So the case for something like CAISI is strong, and dismissing it as a corporate handshake misreads what's actually happening inside those classified rooms.

Now the harder question. What happens when a voluntary evaluation regime is bundled, by the same administration, with a procurement budget that has exploded and an executive willing to use blunt coercive tools against dissenters?

The Pentagon's FY2026 budget request includes $13.4 billion for AI and autonomous systems⁴ — the first standalone AI line item in DoD history, and roughly seven times what was allocated the year before. The single largest beneficiary of the procurement consolidation around that money is Palantir, which in July 2025 received an Army Enterprise Agreement⁵ folding 75 separate contracts into one vehicle worth up to $10 billion over a decade. Palantir's Maven Smart System has, in parallel, been formalized as a program of record⁶ with multi-year funding, after starting from a $480 million ceiling in 2024.

None of that is CAISI's doing. The contracting authority is entirely separate, and that's the standard rebuttal to capture claims: Palantir's primacy predates the May 5 MOUs and runs through DoD acquisition channels CAISI cannot touch. True, but incomplete. The capture story isn't that CAISI hand-picks integrators. It's that the regime creates a two-tier structure in which model providers bear the (currently soft, potentially hardening) costs of cooperation with a federal evaluator, while the integrator that actually fuses those models into kill chains faces no equivalent regime at all. Compliance falls on the layer that has competition; integration revenue flows to the layer that doesn't.

Then there is the Anthropic episode, which is where the "voluntary" framing breaks down. In late February, after Anthropic refused Pentagon demands to drop its prohibitions on autonomous weapons and domestic mass surveillance, Defense Secretary Pete Hegseth designated the company a "supply chain risk" — a label previously used only for entities tied to foreign adversaries like Huawei⁷. President Trump ordered every federal agency to stop using Claude. Within 24 hours, OpenAI signed a Pentagon deal on substantially the terms Anthropic had refused.

A federal judge, Rita Lin, enjoined the designation in March⁸, writing that "nothing in the governing statute supports the Orwellian notion that an American company may be branded a potential adversary and saboteur of the U.S. for expressing disagreement with the government." She found likely First Amendment retaliation. Anthropic claimed the episode had already cost it $180 million in collapsed deals⁹. Microsoft, which has invested $5 billion in Anthropic, filed an amicus brief calling the designation "unprecedented overreach."

Supporters of the current posture make a clean separability argument: the supply-chain designation was a DoD procurement action, not a CAISI evaluation action, and it received judicial review (and lost). The systems are operationally distinct. Anthropic kept its CAISI MOU even as it lost the contract. So the Anthropic case actually proves the regime is not a unified coercive machine.

I find that argument formally correct and substantively unconvincing. Rational firms do not parse authority chains that way. When the same administration controls model evaluation access, procurement decisions worth billions, and a supply-chain designation power that a federal judge had to enjoin to prevent "corporate murder," the labs see one bundle, not three tracks. That is exactly why three companies signed within the same week, and exactly why xAI reportedly agreed unconditionally¹⁰ while Anthropic, holding firm on guardrails, got branded a national security threat. The fact that one of these tools required judicial intervention to stop is not reassurance. It is the capture mechanism operating in plain view.

The cleanest evidence is who's at the table. All three May 5 signatories are closed-weight U.S. incumbents. No major U.S. open-weights developer — Meta, AI2, an American Mistral entity — has signed. Defenders point out, fairly, that this is a choice, not exclusion: Meta has built its own safety stack around Llama 4¹¹ with Llama Guard, CyberSecEval, and Prompt Guard, and has ideological reasons to avoid classified workflows. Fine. But notice the asymmetry the structure produces: incumbents inside the tent shape methodology, get classified feedback loops, and accumulate the soft legitimacy that DoD procurement officers will eventually treat as a baseline. Outsiders get evaluated, if at all, the same way CAISI evaluates Chinese open-weights models — as risk surfaces, not partners. That is the textbook shape of capture, and it does not require fees or licenses to operate. Stigler's original 1971 formulation was always about who shapes the rules, not who pays the toll.

The strongest counter-case is the counterfactual. Congress has had three years to legislate on frontier AI and produced essentially nothing. The realistic alternative to reversible MOUs is not an idealized statute with sunset clauses and APA review; it is either an unregulated frontier or, worse, a statutory licensing regime written by the same incumbent lobbyists who would benefit most from it. Reversibility is a real virtue. A future administration can dissolve CAISI; it cannot easily repeal a licensing law.

I take that seriously. But "the alternatives are worse" is a defense of doing something, not a defense of doing this in this way. The procedural defect is not curable by pointing at a worse hypothetical. A sub-200-person Commerce office, operating without statutory authority, that has become the de facto gatekeeper of what counts as a "safe" frontier model for federal purposes, while the same administration wields supply-chain designation power against firms it dislikes — that is unreviewable executive discretion dressed in the language of measurement science. The right response is congressional action with real procedural guardrails, not continued drift.

What to watch. Three indicators will tell you whether this is capacity-building or moat-building. First, does any major U.S. open-weights developer sign a CAISI MOU on materially the same terms as the closed-weight incumbents within the next 12 months? If not, the asymmetry is structural, not voluntary. Second, does the rumored executive order converting voluntary pre-release review into mandatory pre-clearance¹² actually materialize? That would be the moment the licensing regime stops being de facto and becomes de jure, without a single congressional vote. Third, watch the appeal of Judge Lin's Anthropic ruling. If the supply-chain designation authority survives, the Pentagon retains a coercion tool that makes every "voluntary" agreement adjacent to it less voluntary than it appears. The headlines will keep chasing Musk and copyright suits. The actual regulatory architecture for frontier AI is being built somewhere quieter, and it is being built right now.