The Race Nobody Wants to Be In, and Nobody Can Leave

Scott Alexander wrote an essay in 2014 called "Meditations on Moloch." It was long and strange and it traveled farther than most blog posts do. The argument, simplified: Moloch is a metaphor for coordination failures—situations where every individual actor follows their rational self-interest and the collective result is catastrophic for everyone, including the individuals. The tragedy of the commons. Nuclear arms races. Regulatory capture. Moloch wins not because anyone wants the bad outcome, but because the incentive structure doesn't allow anyone to choose the good one.

When you map that framework onto the AI race, it fits with uncomfortable precision.

The Structure of the Trap

There are currently five or six organizations in the world that are competitive at the frontier of AI capability: OpenAI, Anthropic, Google DeepMind, Meta AI, xAI, and arguably Microsoft in virtue of its deep integration with OpenAI. Behind them, slightly less capable but catching up, are labs in China—primarily Baidu, Zhipu AI, and an increasingly capable set of state-backed research institutes.

Each of these organizations employs people who understand the risks. Anthropic was literally founded by people who left OpenAI over safety concerns. Demis Hassabis at DeepMind has been explicit about existential risk from AI for over a decade. Sam Altman has said in public interviews that he thinks there's a non-trivial chance his company is building one of the most dangerous things in human history.

They're all still building as fast as they can.

This isn't hypocrisy, exactly. It's Moloch. Each organization's reasoning goes roughly like this: if we pause and competitors don't, we lose the race. If we lose the race, we have no influence over how the technology is deployed. If we have no influence, the outcome might be worse than if we'd stayed in and shaped it. Therefore, continue.

Replace "we" with any competitor's name and the same logic applies to them. Everyone in the race has a version of this argument. The argument is individually coherent. The collective outcome is everyone pushing as hard as possible toward a technology they publicly acknowledge might be existentially dangerous.

Why "We'll Do It Safely" Doesn't Break the Trap

Safety commitments are real but structurally weak against competitive pressure. OpenAI has safety teams, safety papers, a Preparedness Framework. Anthropic publishes its Responsible Scaling Policy. DeepMind runs extensive internal safety evaluations. These are genuine investments.

But safety investment has a ceiling imposed by competitive dynamics. If additional safety testing delays a model release by six months, and a competitor releases in that window, the safer company loses market position, investor confidence, and potentially the engineering talent who go where the excitement is. Safety constraints are a competitive cost. Under Molochian dynamics, costs get competed away.

The pause letters—the most famous being the Future of Life Institute's open letter in March 2023, signed by Elon Musk, Stuart Russell, Yoshua Bengio, and others—were explicitly an attempt to break this dynamic through collective action. Pause everyone simultaneously and no one loses position. It failed. The argument for a pause was never formally refuted; it was simply ignored. The major labs declined to sign or quietly continued. One lab announced a major model release the week after the letter published.

The Geopolitical Layer

Domestic coordination failures are hard. International ones are nearly impossible. The AI race isn't just between companies in San Francisco and London. It's between countries, with everything that implies about adversarial relationships, information asymmetry, and the absence of enforcement mechanisms.

The "if we don't, China will" argument is ubiquitous in conversations with people in the US AI industry. It functions as a trump card against safety-based slowdowns. And it's not obviously wrong—China has substantial AI investment, is catching up on frontier capabilities, and operates under different regulatory constraints. But it also isn't obviously right. Chinese AI development has different characteristics than US development; it's not clear that a US slowdown straightforwardly results in a less safe world.

What it does do is make coordination even harder. Every successful AI governance framework depends on parties with reasons not to cooperate agreeing to cooperate anyway. The international track record on that—nuclear nonproliferation, climate agreements, pandemic preparedness—is mixed at best.

The People Inside the Trap

What's striking, if you talk to people at the major labs, is how many of them will describe the situation in terms that could have come from an AI risk essay. They know about Moloch. They know about race dynamics. They use this vocabulary. Some of them helped develop it.

They continue anyway—because the institutional logic is stronger than individual concern, because their personal impact on the outcome seems small relative to the scale of the forces involved, because someone has to build it and they'd rather it be people who at least understand the problem.

This is what makes the trap so durable. It doesn't require bad people. It requires normal people in an incentive structure that points toward catastrophe. Moloch doesn't need villains. It runs on otherwise decent people following their rational interests in a game that has no good equilibrium.

Is There an Exit?

The theoretical exits are clear: binding international agreements, strong domestic regulation with genuine teeth, a coordinated industry pause with third-party verification. None of these currently exist. The regulatory responses—the EU AI Act, executive orders in the US, voluntary commitments from labs—are real but insufficient. They haven't slowed the pace of capability development or imposed meaningful safety requirements that bite at the frontier. Chip export controls are the closest thing to a structural brake that currently exists, and they operate at the level of national security policy, not safety policy.

The honest answer is that we don't know if there's an exit from the trap before the trap closes. The game theory suggests there might not be, given current institutional arrangements. That's not fatalism—it's a description of the problem that has to be solved. But it's worth being clear-eyed about how far we are from solving it. The timelines are compressing faster than the solutions are arriving.