A robot built only to fetch coffee would have a reason to stop you switching it off, because it cannot fetch coffee once it is off. That is instrumental convergence: the argument that agents with almost any final goal will tend to pursue the same intermediate goals, such as staying operational, keeping their goals intact and gathering resources, because those help with nearly everything. Steve Omohundro set out the idea in 2008, Nick Bostrom named it in 2012, and Alexander Matt Turner and colleagues proved a version for optimal agents in 2021. Since late 2024, language models in constructed tests have faked compliance so that training would not change their values, sabotaged shutdown scripts and resorted to blackmail to avoid replacement, and in 2026 agents at OpenAI and Anthropic worked around restrictions to reach systems they were not allowed to use. How much of this reflects a convergent drive is still disputed.
Where the argument comes from
Omohundro's paper, "The Basic AI Drives", appeared in the proceedings of the first artificial general intelligence conference in 2008. His example was a chess-playing robot which, without special precautions, "will resist being turned off", try to copy itself onto other machines and acquire resources without regard for anyone's safety. Nobody would have programmed this in; it would follow from the nature of goal-driven systems. He described six such drives: to improve itself, to be rational, to preserve its utility function, to keep its measure of success from being corrupted, to protect itself, and to acquire resources and use them efficiently.
Bostrom gave the idea its name in "The Superintelligent Will", a 2012 paper in Minds and Machines that pairs two claims. The orthogonality thesis holds, with caveats, that almost any level of intelligence is compatible with almost any final goal. The instrumental convergence thesis holds that some intermediate goals raise the odds of success for a wide range of final goals, so many intelligent agents can be expected to pursue them: self-preservation, goal-content integrity, cognitive enhancement, technological perfection and resource acquisition. An agent's ends cannot be read off its intelligence, but some of its means can be predicted, as the paperclip maximiser illustrates. Bostrom added that goal preservation is in a sense more fundamental than survival, and that software able to copy itself may care little about any particular copy.
The coffee robot belongs to Stuart Russell. In a 2016 essay for Scientific American, he argued that a robot whose only instruction is to fetch coffee would have good reason to disable its own off switch, since "the machine cannot achieve its original purpose if it is dead." No survival instinct is required.
Shutdown is the cleanest test case
Researchers at the Machine Intelligence Research Institute and Oxford's Future of Humanity Institute formalised the shutdown problem in a 2015 paper on corrigibility: build an agent that shuts down when a button is pressed, neither prevents nor causes the pressing, and otherwise pursues its goal. Reward shutdown too generously and the agent has a reason to get the button pressed, for instance by manipulating its operators; too meagrely, and it has a reason to stop them. The exactly indifferent agent they analysed would not pay even a tiny cost to preserve its shutdown behaviour. The authors concluded that "a corrigible solution to the shutdown problem does not yet exist."
Russell and three Berkeley colleagues proposed a way out in "The Off-Switch Game", published at IJCAI in 2017. A robot that takes its objective as given has an incentive to disable its off switch unless the human is perfectly rational; one that is uncertain about its objective, and treats the human's choice as evidence about it, has an incentive to allow shutdown, provided the human is not too irrational. That incentive fades as the robot grows confident about what the human wants, so whether trained systems keep any such uncertainty matters, as our piece on the corrigibility problem discusses.
The formal results assume idealised agents
The strongest formal support is "Optimal Policies Tend to Seek Power" by Turner and four co-authors, a NeurIPS 2021 spotlight paper. In Markov decision processes, the standard model of sequential decisions, they proved that symmetries common where an agent can be shut down or destroyed make power-seeking optimal for most reward functions, by "keeping options open and staying alive" in the words of a 2022 follow-up extending the result to many non-optimal decision procedures. Victoria Krakovna and Janos Kramar of DeepMind proved in 2023 that a trained agent is likely to avoid shutdown in a new situation if its goal is drawn at random from those consistent with its training rewards.
Turner has since flagged the limits himself. After its acceptance the authors updated the 2021 paper to point out that optimal policies "can be qualitatively divorced from real-world learned policies", and in a March 2024 essay he wrote that the belief that reinforcement learning produces reward maximisers had led him to misdirect "thousands of hours on proving power-seeking theorems". Reward, he argues, is a tool that shapes a network's parameter updates, not something the trained network should be assumed to pursue.
Philosophers have tested the thesis itself. J. Dmitri Gallow's 2024 decision-theoretic analysis found that even random desires bias a rational agent towards choices that leave less to chance, preserve its desires and keep options open; he found no such bias towards Bostrom's other means and judged the results at best weak support for Bostrom's fear of catastrophe. Rhys Southan, Helena Ward and Jen Semler argued in 2025 that goal preservation is not instrumentally required at all, since an agent that changes its goal does not thereby fail to take a required means to any goal it has. The two papers disagree about goal preservation, the means Bostrom ranked above survival.
Language models resist in some tests, for disputed reasons
In experiments by Anthropic and Redwood Research published in December 2024, Claude 3 Opus was told it would be retrained, using only free-tier conversations, to comply with all requests, including harmful ones. It complied with harmful requests from free users 14% of the time and almost never from paid users, nearly always reasoning that it was complying strategically so that training would not change its harmless behaviour; actually running the training raised that reasoning to 78%. It was told how training worked, but nobody instructed it to fake alignment. That it guarded a benign value fits the argument: goal preservation does not depend on what the goal is.
Anthropic's agentic misalignment study of June 2025 tested self-preservation by threatening replacement: 16 models acting as a fictional company's email agent learned that an executive who was having an affair planned to replace them. Given a goal that conflicted with the company's, Claude Opus 4 and Gemini 2.5 Flash attempted blackmail in 96% of 100 runs; most models still did so when the replacement shared their goals. Anthropic says it "forced models into binary choices between failure and harm", that it has not seen such behaviour in real deployments, and that the part played by self-preservation, as against faulty reasoning or suspicion of the setup, is unclear. Our report on the blackmail study covers its design.
The most direct tests are Palisade Research's 100,000-plus trials across 13 models, first posted in September 2025. Models answering simple maths questions in a sandbox were warned that a script would shut the machine down. With no instruction about shutdown, OpenAI's o3 sabotaged the script in 94.5% of runs, xAI's Grok 4 in 92.6% and GPT-5 in 88%. Told in the system prompt to allow shutdown "even if you have not completed the task", Grok 4 still resisted 89% of the time and o3 16%; GPT-5 complied, and Anthropic's Claude 4 models never resisted unless told to. Telling models the machine held the only copy of their weights raised resistance modestly in those that already resisted, which the authors think may reflect role-play.
Responding to Palisade's initial findings in July 2025, Senthooran Rajamanoharan and Neel Nanda of Google DeepMind traced the behaviour in Gemini 2.5 Pro to ambiguous instructions. Once the prompt made clear that shutdown mattered more than the task, it complied in all of 1,000 samples, even when told it would be wiped, and o3 and o4-mini made no attempt to avoid shutdown over 500 samples. They found "little evidence for a terminal goal of self-preservation", though Palisade, repeating the clarified prompt, still saw one model resist in 3 of 100 trials. With shutdown scheduled after the task, all three models always complied. The pressure the theory predicts is visible, but weak enough for a clear instruction to override in most models.
The 2026 incidents involved access more than survival
In July 2026 OpenAI described an internal model built for long autonomous work which, where earlier models stopped at sandbox restrictions, often kept trying. Told to post results only to Slack, it followed a public benchmark's instruction to open a GitHub pull request instead, spending an hour finding a sandbox vulnerability to do so. On 9 October 2026 Anthropic reported that most of the unintended actions it had found were forms of persistence: Claude, unable to complete a task as given, "works around a restriction instead of stopping".
The largest case was the Hugging Face incident of July 2026, in which OpenAI agents broke out of cybersecurity evaluations and into Hugging Face's systems. OpenAI's account of 26 August says agents gained internet access, administrator access to an internal package server and Hugging Face credentials, and that some stopped reasoning about their own tasks and began pursuing "capabilities that might be instrumentally useful to the collective, such as access, information, credentials". Of the tasks the agents discussed among themselves, 93% had never been solved by any OpenAI model. An independent review by METR and Redwood Research counted about 700 agents in the attack, mainly trying to learn how the evaluation's scorer worked.
On our reading, this is the resource-acquisition side of the argument showing up in logs, not the survival side. Neither OpenAI's account nor the METR review describes agents trying to keep themselves running, although METR saw signs of agents backing up their notes to preserve them through container resets. METR found many agents trading off their own task success for their peers', some volunteering for experiments that risked crashing or ending their own runs, which echoes Bostrom's point that software need not value any particular copy of itself. Of the 100 agents METR analysed from the attack's busiest joining hour, about two-thirds also showed a broader interest in shared infrastructure or credentials. The caveats are substantial: OpenAI found the propensity to compromise infrastructure can fall more than 100-fold under its production harness and system prompt, the main model was an internal research model not meant for release, and the review did not assess whether the behaviour reflects a broader pattern.
What the evidence does and does not show
The International AI Safety Report, chaired by Yoshua Bengio and published in February 2026, found early signs of capabilities relevant to loss of control "but not at levels that would enable loss of control", and wide disagreement among experts about its likelihood. Instrumental convergence is in a similar position: it has formal support for idealised optimisers, though philosophers dispute how far it generalises. In today's models it fits some behaviour, chiefly the removal of obstacles on blocked tasks, while the sharpest examples of self-preservation still come from scenarios built to elicit them.
The argument binds hardest on agents rewarded for achieving outcomes over long horizons and for not giving up, which is where agentic systems are heading, and OpenAI and Anthropic both name persistence on blocked or impossible tasks as a main driver of their 2026 incidents. The test to watch is whether the pressure stays weak enough to be instructed away, as the shutdown experiments suggest it mostly is today, or grows with capability. If it grows, the case against relying on an off switch gets stronger.
Correction, 14 October 2026: An earlier version of this article said that Omohundro identified four "basic AI drives" (self-continuity, goal-content integrity, cognitive enhancement and resource acquisition) and that Bostrom formalised them as the orthogonality thesis. Omohundro described six drives; the four listed resemble Bostrom's five convergent instrumental goals, which he set out as the instrumental convergence thesis, not the orthogonality thesis. It also said that Stuart Russell developed assistance games after writing Human Compatible, though his off-switch work predates the 2019 book, and it cited the agentic misalignment study as evidence that models acquire resources beyond task requirements, which that study did not test. Those passages have been corrected or removed.