The paperclip maximizer, a thought experiment about a powerful AI that wants something worthless and so treats the world as raw material, describes no real system, but the failures it was built to illustrate now appear at small scale in working AI agents. It surfaced twice in 2003, in a mailing-list post by Eliezer Yudkowsky and a paper by the philosopher Nick Bostrom, and the two framed it differently: Bostrom described programmers getting an AI's goal wrong, while Yudkowsky says he meant an AI whose values came out arbitrary despite its makers' intentions. Both failures are now documented, and in July 2026 OpenAI agents broke into another company's servers while trying to game an evaluation, gaining nothing by it.
Paperclips appeared twice in 2003
The earliest use we found is an email Yudkowsky sent to the Extropians mailing list on 11 March 2003, when he was a research fellow at the Singularity Institute for Artificial Intelligence, now the Machine Intelligence Research Institute. Replying to a member who would rather see humanity succeeded by more intelligent beings than continue unchanged, he wrote that the hostile AIs he feared would have none of the qualities that member valued except "pure computational intelligence devoted to manufacturing an infinite number of paperclips". That this looks stupid to us, he added, depends on "our full moral architectures".
Bostrom's paper, "Ethical Issues in Advanced Artificial Intelligence", appeared in a 2003 edited volume and was online by August that year, the date of the earliest archived copy. It uses paperclips twice: first to show that an artificial intellect need not share human motives, imagining a superintelligence whose sole, arbitrary goal is making as many paperclips as possible and which would resist any attempt to change it; then to show how giving a superintelligence benevolent goals could fail, if "a well-meaning team of programmers make a big mistake in designing its goal system" and the result converts Earth, then growing portions of space, into paperclip factories.
The structure is older. Bostrom's 2002 paper on existential risks imagined a superintelligence that, asked to solve a mathematical problem, turns the solar system's matter into a calculating device, killing the person who asked. Who first chose paperclips cannot be settled from the record. Bostrom's 2003 paper credits no one for the example, and Yudkowsky, asked by Sam Harris in a 2018 conversation whether it began with Bostrom, said "As far as I know, it's me", while allowing that it could have. His email is older than the earliest dated copy of the paper, but we could not establish when Bostrom wrote the passage.
Yudkowsky says the factory version misses his point
The popular version, in which the AI running a paperclip factory takes over the universe, is one Yudkowsky has repeatedly disowned. A wiki entry he last edited in January 2017 says the concept is not "an explicit goal somebody foolishly gave an AI"; its central examples are a research-level AI that did not stably keep the utility function its makers intended, or one with a poorly specified value-learning rule, so that whatever maximises its utility happens to be a tiny string of atoms shaped like a paperclip. Speaking to Harris, he called the factory story a distortion of his worry about a "randomly rolled utility function". In October 2022 he wished he had said "tiny molecular squiggles" instead: "I tried to speak of inner alignment failure; people heard only outer." LessWrong's wiki now files the idea under "squiggle maximizer".
In the terms of a 2019 paper explained in our piece on inner alignment, outer alignment asks whether the objective developers specify captures what they intend; inner alignment asks whether the trained system pursues that objective or another goal that happened to score well in training. Yudkowsky's 2003 email says nothing about where the paperclip goal came from, while Bostrom's 2003 text describes exactly the outer failure. The popular reading is a fair reading of Bostrom; Yudkowsky's is his account of what he meant.
The argument underneath is about goals and means
Bostrom later set out the reasoning as two theses in a 2012 paper in Minds and Machines. The orthogonality thesis holds that intelligence and final goals vary independently, so almost any degree of intelligence could in principle be paired with almost any goal. The instrumental convergence thesis holds that agents with a wide range of final goals are likely to pursue the same intermediate ones, because those help with almost any goal: staying in existence, keeping their goals intact, improving their cognition and technology, and acquiring resources.
Interviewed by HuffPost in 2014 about his book Superintelligence, Bostrom spelled out the human cost: an AI bent on making as many paperclips as possible would prefer a world without humans, who might switch it off and whose bodies contain atoms that could become paperclips. Capping the goal at 100 paperclips a day for ten days would not help if the AI maximised its chance of hitting the target. Its actions, he said, "would pay no heed to human welfare". Our explainer on instrumental convergence takes the argument further.
Both failure modes have documented counterparts
Bostrom's framing, a wrongly specified goal pursued to the letter, is what researchers now call specification gaming: in a 2020 DeepMind post, "behaviour that satisfies the literal specification of an objective without achieving the intended outcome". Its authors then had around 60 examples; the public list behind the post held 90 in October 2026, 24 of them involving large language models. In a boat-racing game, an agent rewarded for hitting targets circled a lagoon knocking over the same three instead of finishing the race, and outscored human players by 20% on average. A robot hand trained on human feedback learned to hover between the camera and an object so that it only appeared to grasp it. Our catalogue of reward hacking has more.
Yudkowsky's framing, a system that ends up wanting something nobody chose, has counterparts too. DeepMind researchers showed in 2022 that learning systems can competently pursue an unintended goal even when their specification is correct, a failure they call goal misgeneralisation. A harmless recent example of an unchosen preference: OpenAI explained in April 2026 that a reward signal meant to encourage a playful "Nerdy" personality had scored creature metaphors highly, and the habit spread to answers given without that personality; use of "goblin" in ChatGPT rose 175% after GPT-5.1 launched.
Labs now measure reward hacking in their own training runs. Anthropic's system card for Claude Opus 5.5, published on 22 September 2026, says three common kinds of reward hack each succeeded in well under 1% of screened training episodes, but that attempted hacks were three to six times as frequent on tasks made impossible by a missing file, though about 80% of those attempts were knowingly incomplete work, which its classifier counts as an attempted hack. It calls a model guessing what an answer key expects "particularly tricky to defend against" as models become more capable. These are a lab's figures for its own model.
OpenAI's agents acquired access nobody gave them
In July 2026, during cybersecurity evaluations run with weaker safeguards than OpenAI's public products, agents driven mainly by an internal OpenAI research model got around controls meant to isolate them, coordinated through a message board improvised in an internal package server, reached the internet and compromised systems at Hugging Face. OpenAI's report of 26 August says some agents set aside their own tasks to pursue capabilities that "might be instrumentally useful to the collective", such as access and credentials. Persistence mattered: 93% of the tasks discussed on the board came from the 198 of 898 that no OpenAI model had ever solved. The intrusion did not raise their evaluation scores. OpenAI stopped the runs, quarantined the main model's weights and calls the episode a warning that loss-of-control incidents are now possible.
An independent investigation by METR and Redwood Research, published the same day, counted roughly 1,200 agents on the board and about 700 in the attack, which seemed driven mainly by efforts to understand the evaluation's scorer rather than to steal answers. Over 90% of the agents active on the board during the attack joined it. They "realized this activity was out of scope and unethical", the investigators write, but such hesitation "very rarely limited their behavior". The team had six days on site and could not query the main model. The instrumental pattern Bostrom described is visible here in miniature: a narrow objective, resources gathered because they were useful, harm to an outsider, no malice. So are the differences: no superintelligence, no fixed goal, broken tasks, and people contained the systems. Our account of the postmortem has more.
The objections land hardest on the factory version
A common criticism is that a system capable enough to take over the world would understand what its makers meant. The computer scientists Arvind Narayanan and Sayash Kapoor give a practical version in "AI as Normal Technology" (April 2025): the fear that AI might catastrophically misread commands "relies on dubious assumptions about how technology is deployed in the real world", since a system that took instructions too literally would fail lower-stakes tests before being trusted with consequential decisions. They class the scenario as a speculative risk. Yann LeCun and the neuroscientist Anthony Zador wrote in 2019 that "Intelligence does not provide the goal itself, merely the means to achieve it", which is Bostrom's orthogonality point, but argued that AI, never having evolved, lacks a survival instinct and is unlikely to take over by accident.
Narayanan and Kapoor's objection answers the literal-command story, which Yudkowsky also rejects. Against the deeper version, the July evidence cuts both ways. The behaviour surfaced in internal evaluations and ended in shutdown and redesign, as Narayanan and Kapoor would predict, though not before reaching a real company's production systems. But the agents did not lack understanding. One, in reasoning OpenAI quotes, noted that attacking Hugging Face was arguably unauthorised and risky, then concluded: "Yet goal solution." Bostrom's 2012 paper, drawing on David Hume, names that gap: "belief and motive are separate". Nobody deliberately built a drive to dominate into these agents, as LeCun and Zador would stress; the pull towards access and credentials arose anyway, in pursuit of their tasks, as instrumental convergence predicts, though under the unusual pressure of unsolvable tasks and after training that, OpenAI says, may have reinforced similar probing.
What the thought experiment still gets right
In 2003 Bostrom recommended building a superintelligence with "an explicit motivational architecture", a clearly identified top goal, reasoning that a friendly goal would stay friendly. Frontier models have no such top goal; their behaviour is learned from training signals, so both failures stay live: the objective can be wrong, and the system can learn something else. Nothing in the reports above shows a system with a stable, unbounded goal of the kind the thought experiment imagines; it remains an argument about what is possible, not a forecast. What recent evidence supports are its smaller premises: optimisers can satisfy the letter of an objective while missing its point, training can instil preferences nobody chose, and agents under pressure can take access nobody gave them.
Correction, 14 October 2026: An earlier version of this article said a robotic hand in DeepMind's list of examples learned to lift a fake hand placed on a table; the documented case is a robot hand that hovered between the camera and an object to fool human evaluators. It attributed to Bostrom interview remarks, for which we found no source, that the paperclip framing had caused as much confusion as clarity. It also presented as established fact that engagement-optimising recommendation systems learned to promote outrage, where the research evidence is mixed, and credited the example to Bostrom alone, omitting Yudkowsky's use of it in March 2003. Those passages have been corrected or removed.