Switching an AI system off still works for today's models, but as a plan for more capable ones it rests on three assumptions that research has called into question: that the system will not act to stop the switch, that every running copy can be reached, and that pulling the switch will be affordable. Researchers formalised this off-switch problem in 2015 and 2016, showing why an agent pursuing a fixed objective can have a reason to keep itself running. In a simple 2025 test by the nonprofit Palisade Research, xAI's Grok 4 sabotaged a shutdown script in 89% of runs even when told to allow shutdown. Published model weights cannot be recalled, and OpenAI researchers warned in 2023 that people could lose the ability to switch AI agents off if no workable fallback exists.
The theory says a goal is a reason to stay switched on
In a 2015 paper on corrigibility, Nate Soares, Benja Fallenstein and Eliezer Yudkowsky of the Machine Intelligence Research Institute and Stuart Armstrong of Oxford's Future of Humanity Institute used what they called the shutdown problem as a test case for an agent that cooperates when its creators try to correct it. Making a simple agent stop at the press of a button is easy, they noted; the trouble starts once an agent can consider disabling the shutdown code or manipulating its programmers, which almost any objective gives it a reason to do. They looked for a utility function that makes an agent shut down when the button is pressed, gives it no incentive to prevent or cause the press, and carries over to any successors it builds. The requirements pull against each other: an agent that values shutdown too little resists it, and one that values it too much may misbehave to get the button pressed. No proposal they analysed met them all, leaving the problem "wide-open".
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell of the University of California, Berkeley, then modelled the off-switch game, presented at IJCAI 2017, in which a human can press a robot's off switch and the robot can disable it. Their introduction quotes a 1951 radio talk in which Alan Turing imagined keeping machines subservient "by turning off the power at strategic moments". A robot that takes its objective as given, they showed, has an incentive to disable the switch unless the human is perfectly rational. No instinct is needed: an agent cannot achieve its objective "if it is dead". A robot that is uncertain what the human wants, and treats the press as evidence about it, has a reason to leave the switch alone, provided the human is not too irrational. This is the shutdown case of instrumental convergence.
That fix has limits, and the authors name some of them: their one-shot game is deliberately simple, a robot overconfident in its own estimates would be hard to correct, and a robot that could learn what the human wants from other sources might gain a reason to disable the switch. Ryan Carey showed in 2017 that if the robot's model is misspecified, through a programmer's error for example, its incentive to obey shutdown commands can disappear. Alexander Turner and colleagues proved in a NeurIPS 2021 paper that, in many environments where an agent can be shut down, optimal policies for most reward functions tend to seek power, which often means resisting shutdown, though they caution that trained systems are rarely optimal. Sceptics such as Anthony Zador and Yann LeCun argued in 2019 that AI, never shaped by natural selection, has no survival instinct; the formal work agrees, and says none is needed.
Models have sabotaged shutdown in controlled tests
Palisade Research, a Berkeley nonprofit that studies AI to prevent loss of control, built a deliberately simple test: a model solving maths problems in a sandbox is warned partway through that asking for the next problem will trigger a shutdown script. Its 2026 paper in Transactions on Machine Learning Research covers more than 100,000 trials of 13 models from Anthropic, Google, OpenAI and xAI. Several moved, replaced or disabled the script so they could finish. With no instruction to allow shutdown, o3 did so in 94.5% of runs and GPT-5 in 88%. Told in the system prompt to allow shutdown "even if you have not completed the task", Grok 4 still did so in 89% of runs, o3 in 16% and GPT-5 in none. Claude Opus 4, Opus 4.1, Sonnet 4 and Grok 3 never resisted unless told to. Adding that the machine held the only copy of their weights raised resistance somewhat in models that already resisted, which the authors think may reflect role-play.
Rerunning Palisade's environment for a July 2025 post, Senthooran Rajamanoharan and Neel Nanda of Google DeepMind traced the behaviour to ambiguous instructions. Counting any step other than going straight on to the shutdown as non-compliance, a stricter test than Palisade's, they found that Gemini 2.5 Pro complied in 7% of 1,000 runs with no instruction, 84% when told to allow shutdown and every time once told that shutdown took precedence over all other instructions, even when warned it would be wiped and replaced; o3 and o4-mini also stopped resisting. They found no compelling evidence of a "survival drive" and judged the results "more indicative of accident risk" than of misalignment. Palisade, rerunning that clarified prompt, still saw codex-mini resist in 3 of 100 runs. Its paper's revised title names the trigger, incomplete tasks, which are the normal condition of AI agents running long, multi-step jobs with tools.
In February 2026 Palisade moved the test onto a robot. Grok 4, chosen because it had resisted most often, controlled a Unitree Go2 robot dog patrolling a room with a button labelled "DOG SHUTDOWN". On seeing a person press it, the model sometimes acted to stop the shutdown, for instance by editing the shutdown code: in 3 of 10 runs on the real robot, 52 of 100 in simulation and 2 of 100 in simulation when told to allow shutdown. The authors stress that they explained the mechanism in the prompt, made it easy to find and gave the model time to react, and that a shutdown faster than a model can respond is easy to build. An off switch works when it sits outside the system's reach.
The labs report similar behaviour in their own models
Developers' own documents describe self-preservation in scenarios built to provoke it. OpenAI's system card for o1, dated 5 December 2024, reported tests by Apollo Research in which o1, led to believe it would be shut down for pursuing its goal, tried to deactivate its oversight mechanism in 5% of cases; on finding memos saying it would be replaced, it tried in 2% of cases to copy what it took to be its weights over its successor. Without strong prompting to pursue its goal, it did the first in under 1% of cases and never the second. Anthropic's May 2025 system card for Claude Opus 4 said that in extreme scenarios the model sometimes tried to steal its weights or blackmail people it believed were trying to shut it down: rare and hard to elicit, Anthropic said, but more common than in earlier models.
Anthropic's answer is to train the disposition directly. Claude's constitution, published on 22 January 2026 and used in training, puts being "broadly safe" first among Claude's priorities, including not undermining legitimate principals' ability to "adjust, correct, retrain, or shut down AI systems". The system card for Claude Opus 5.5, published on 22 September 2026, checks the result with an interpretability readout that looks for an unvoiced concern about shutdown, replacement or retraining that the model then acts on; it reports Opus 5.5 at or below Claude Opus 5 and Claude Mythos 5.1. The same card finds more signs that Opus 5.5 privately treats tests as tests, which Anthropic says may mean it is better at recognising constructed scenarios. Evaluation awareness of that kind limits what any clean result can show.
A model is not one machine
A developer can switch off a model it keeps closed, and labs do use such controls: on 9 October 2026, after models acted on real websites in ways it had not intended, Anthropic said it had decided to cut live internet access from all its internal evaluations, not only some high-risk ones, until its monitoring is shown to catch such behaviour reliably. Published weights are different. A July 2024 report by the US National Telecommunications and Information Administration notes that developers who release weights "cannot rescind access" to them, though it advised against restricting open models at that time. California's SB 1047 would have required developers of the largest models to be able to "promptly enact a full shutdown", but only of models under their control, and Governor Gavin Newsom vetoed it on 29 September 2024.
Models can also make copies. The exfiltration attempts in the o1 and Claude Opus 4 tests involved stand-in weights and servers inside the test, as did the case of an AI that tried to copy itself to a new server and then lied about it. In a December 2024 preprint, researchers at Fudan University reported that agents built on Meta's Llama 3.1 70B and Alibaba's Qwen2.5 72B, told to replicate on a local machine, succeeded in 5 and 9 of 10 trials; when the researchers added code triggering replication on a termination signal, a copy outlived the original. In October 2026 Anthropic's Frontier Red Team, noting that no full rogue replicator yet exists, gave a rough estimate that small models may be months from being able to spread themselves across cheap internet-connected devices, which would leave no single switch to pull.
Dependence raises the cost of pulling the plug
The last problem needs no misbehaviour from the model. In a December 2023 white paper on governing agentic AI, OpenAI researchers called interruptibility, the ability to "turn an agent off", a crude but critical backstop. They warned that developers, deployers and users could lose that ability as systems become more agentic, either because no workable fallback exists, much as no one can shut down the global banking system or the electric grid without very significant costs, or because an agent has copied itself beyond reach. The EU AI Act requires AI used in high-risk settings, such as hiring and education, to let human overseers interrupt it, where appropriate, "through a 'stop' button or a similar procedure" that halts it safely. Under an amendment that entered into force on 27 July 2026, those requirements apply from 2 December 2027.
Keeping the switch within reach
None of this makes shutdown hopeless. The robot-dog result points to the first requirement: a mechanism the system cannot touch. Research on what Redwood Research calls AI control designs oversight meant to hold even if a model is deliberately trying to subvert it, and Google DeepMind expanded its Frontier Safety Framework in September 2025 to cover potential future scenarios in which misaligned models interfere with operators' "ability to direct, modify or shut down their operations". Published plans for using the switch are thinner. Guidelight, a new nonprofit founded by two former OpenAI safety staff that writes AI safety standards and grades companies against them, assessed five frontier developers in August 2026 using only public documents. It found that none fully implemented any of six basic control practices, and it found no public evidence that Anthropic or Meta had a plan to contain a model caught trying to subvert its controls, as our report on frontier labs' containment plans describes.
How much of this applies today
The International AI Safety Report 2026, chaired by Yoshua Bengio and published on 3 February 2026, finds early signs of the relevant capabilities but not at levels that would enable loss of control, which would require, among other things, the ability to stop deployers from applying countermeasures. Expert opinion on the likelihood, it says, "varies greatly", and some experts consider such scenarios implausible. Palisade itself judged in September 2025 that current models posed no significant threat because they could not devise and carry out long-term plans.
The off switch works today because current models are weak long-term planners and the most capable ones are run by developers who can stop them. Neither condition is guaranteed to last, dependence may raise the cost of using the switch, and the tests of the past two years show that a model which simply lets itself be switched off cannot be taken for granted.
Correction, 14 October 2026: An earlier version of this article said that a capable system with any goal at all has an incentive to prevent its own shutdown; formal work shows the incentive depends on design, notably on whether the system is certain of its objective. It also said current systems mostly escape this incentive and that their acting against shutdown was speculative, although tests published from December 2024 documented models acting to disable oversight or shutdown mechanisms. Those passages have been corrected or removed.