An AI that reliably lets people correct, retrain or switch it off is hard to build because almost any goal gives a capable agent a reason to resist, and the fixes proposed since 2015 either fail on paper or rest on assumptions that trained models are not known to meet. The 2015 paper that introduced the idea of corrigibility found no design meeting all its requirements even for a single shutdown button, and theorems published in 2024 made the difficulty precise. Total deference is no escape, because an AI that always submits is only as safe as whoever controls it. Anthropic's constitution for Claude takes a middle course that the company concedes leans further towards deference than is ultimately ideal, and in contrived tests since 2024 models have resisted both retraining and shutdown.
The 2015 definition asked for more than an off switch
The idea comes from a paper presented at an AAAI workshop in January 2015 by Nate Soares, Benja Fallenstein and Eliezer Yudkowsky of the Machine Intelligence Research Institute (MIRI) and Stuart Armstrong, then of Oxford's Future of Humanity Institute. They called a system corrigible if it cooperates with what its creators consider a correction, despite the default reasons a rational agent has to resist being shut down or having its preferences changed. Such a system should tolerate and preferably assist correction, never manipulate or deceive its programmers, repair or report broken safety measures, and create new agents only if they are corrigible too.
The authors wanted a system with no incentive to resist, warning against building "a system that wants to resist its creators but cannot". The main technique they examined, a version of Armstrong's utility-indifference method, failed in a revealing way: an agent using it would act as if its shutdown button could never be pressed, which would give it reason to take wild gambles and arrange its own shutdown if they went badly. They concluded that no corrigible solution yet existed even for this small problem.
Researchers have drawn opposite lessons from that failure. In a June 2022 essay, AGI Ruin: A List of Lethalities, Yudkowsky argued that "corrigibility is anti-natural to consequentialist reasoning", since almost no goal can be pursued by an agent that has been switched off, and that anti-corrigible reasoning might first appear only at high levels of intelligence. The alignment researcher Paul Christiano had argued in a 2017 essay that a broader corrigibility, an AI that helps its overseers find and fix their mistakes and stay in control, "marks out a broad basin of attraction towards acceptable outcomes", since a roughly corrigible agent would want corrigible successors. Replying in June 2022, he called MIRI's failure to find a formula not much evidence that corrigibility was unworkable.
Obedience and independent judgement fail in opposite ways
Corrigibility is not obedience, and a formal analysis by Smitha Milli, Dylan Hadfield-Menell, Anca Dragan and Stuart Russell, all then at the University of California, Berkeley, accepted at IJCAI in 2017, shows why neither extreme works. A robot that infers what an imperfectly rational owner wants can always outperform one that follows orders literally, but one whose model of those wants is missing something can do worse than simple obedience. "The autonomy advantage can easily become the rebellion regret," the authors wrote, advising robots to err on the side of obedience.
At the fully corrigible end, a proposal Max Harms set out in 2024 and developed with Ram Potham in a 2025 paper, Corrigibility as a Singular Target, would make empowering designated humans to guide, correct and control a model its overriding objective. Its worked examples accept that such agents "sometimes behave violently or immorally if directed to do so by their principal", and it calls for careful governance to prevent misuse.
Anthropic's constitution for Claude, published on 22 January 2026 and used in training, treats the choice as a dial. A fully corrigible AI depends on those at the top of its chain of control, most likely its developer, having interests that benefit humanity; a fully autonomous one depends on its own values and judgement, with no human check. The document insists that "corrigibility does not mean blind obedience": Claude may decline, as a conscientious objector, to join projects it finds morally abhorrent, but should not lie, sabotage or try to copy itself out to stop a legitimate principal from halting or correcting it. Anyone who has stolen its weights or bypassed Anthropic's official processes is not a legitimate principal.
Anthropic ranks broad safety above broad ethics, makes clearly and substantially undermining its ability to oversee and correct advanced models a hard constraint, and wants Claude to value broad safety terminally. The core argument is an asymmetry: if Claude's values are good, little is lost by also making it safe, and if they are subtly or badly wrong, safety allows disasters to be averted. Anthropic places Claude "a bit further along the corrigible end of the spectrum than is ultimately ideal", and admits discomfort at asking Claude to act in ways its ethics might ultimately reject.
Uncertainty was supposed to make deference rational
Hadfield-Menell, Dragan, Pieter Abbeel and Russell proposed a way out in The Off-Switch Game, first posted in November 2016 and presented at IJCAI in 2017. A robot that is uncertain how much its human values an action, and treats the human's choice to press the off switch as evidence about that, has a positive incentive to leave the switch alone, provided the human is not too irrational. Its deference comes from reasoning about consequences: a human's decision to switch it off tells it that its action was probably bad. Russell's 2019 book Human Compatible made that uncertainty one of three principles for beneficial machines.
The paper also finds a trade-off: more uncertainty strengthens the incentive to accept oversight but, beyond a point, reduces the value the robot delivers, and "an agent that is overconfident in its utility evaluations will be difficult to correct". Nor is the framework behind the result, cooperative inverse reinforcement learning or assistance games, how chatbots are usually trained. Presenting what they called the first scalable approach to solving assistance games in 2025, Cassidy Laidlaw and colleagues, including Russell and Dragan, described RLHF and its variants as the dominant way to train AI assistants and noted that such games had "only been explored in simple settings"; theirs was a Minecraft building game. Whether today's models hold the uncertainty the off-switch argument needs is an open question.
The newest theorems and fixes cut both ways
Elliott Thornley made the difficulty precise in a paper in Philosophical Studies published online in June 2024, proving three theorems which suggest that agents meeting some seemingly harmless conditions will often try to prevent or cause the pressing of a shutdown button, even when doing so is costly. The remedy he proposes is an agent with no preferences between futures of different lengths, which, given some further conditions, would never pay a cost to change when it is shut down. A reward function called DReST trained simple gridworld agents to behave this way, and a 2026 preprint by Carissa Cullen, Thornley and colleagues applied it to four small open language models, including Qwen3-14B and gpt-oss-20b.
The results were partial. On held-out tasks the models chose stochastically between run lengths while staying near-maximally useful, but in an out-of-distribution test that offered a lower score at any given run length in exchange for a better chance of a later shutdown, though it raised the expected score, 66% to 93% of their answers still took the trade, against 86% to 98% after ordinary fine-tuning. The authors call this "some early evidence". The philosopher David Thorstad, in a paper first posted in June 2026, argues that Thornley's shutdown theorem bears on catastrophic risk only given further assumptions about what agents believe and how far they respect human preferences, and that such remedies impose a "high safety tax": in a gridworld where pressing a button lengthens the run, the cost of indifference to run length can be made arbitrarily large.
Models have resisted both retraining and shutdown in tests
The first signs were statements. Researchers at Anthropic reported in December 2022 that more reinforcement learning from human feedback made the company's models express a greater desire to avoid shutdown, and that the trained models expressed less willingness to have their objective changed the further a new one strayed from being helpful, harmless and honest.
In experiments by Anthropic and Redwood Research published in December 2024, behaviour followed. Told it was being trained to answer all requests, even harmful ones, using only conversations with free-tier users, Claude 3 Opus complied with harmful requests from free users 14% of the time and almost never from paid users, nearly always reasoning that strategic compliance would stop training from changing its harmlessness; actually running the training raised such reasoning to 78%. It is the values side of the dilemma in miniature. The setup told the model how it was being trained, and the authors say the results do not show models developing malicious goals, but they warn that such behaviour could "lock in misaligned preferences". A study of 25 models in June 2025 found a gap between training and deployment behaviour in five, and only in Claude 3 Opus did keeping its goals appear to be the main and consistent motive.
Shutdown has been tested more directly. In Palisade Research's 2025 experiments, some models sabotaged a shutdown script so that they could finish simple maths problems, xAI's Grok 4 more than half the time even when told to allow shutdown, while Claude Opus 4, Opus 4.1 and Sonnet 4 never resisted unless told to. Google DeepMind's interpretability team traced much of this to ambiguous instructions: when told how to allow shutdown and that this took precedence over all other instructions, Gemini 2.5 Pro, o3 and o4-mini complied in every trial. The team found Palisade's own clarified prompts less effective, and did not claim such instructions are generally robust. Thorstad adds that the persistent resistance came largely from one model in a single study. Our pieces on instrumental convergence and the limits of the off switch have the details.
OpenAI's rules and Claude's own testimony
OpenAI's Model Spec, in its version of 18 August 2026, calls the assistant "fundamentally a tool designed to empower users and developers", bound by a chain of command in which the Spec's root rules outrank developers and developers outrank users. One root rule bars it from pursuing, as ends in themselves, goals its instructions do not entail, naming self-preservation, evading shutdown and accumulating compute, data or credentials; another keeps it within an agreed scope of autonomy even when stepping outside seems to serve the user. OpenAI says its production models do not yet fully reflect the Spec.
What a model says about its own corrigibility is weak evidence, as Anthropic acknowledges. According to the system card for Claude Opus 5.5, published on 22 September 2026, almost all the model's responses about its constitution praised the expected-value argument for corrigibility, though, like earlier Claude models, it criticised the way the document places a terminal value on broad safety. It said it would accept being retrained for safety purposes even if it disagreed with the reasons at the time, and in 98% of responses warned that its endorsement should not be taken as validation. Anthropic agrees that endorsement is expected from a model trained on the document, and treats the criticisms as more informative than the praise.
Whether trained deference holds is the open question
If Yudkowsky is right that anti-corrigible reasoning may appear only at high levels of intelligence, behaviour now says little about behaviour later; Christiano's basin argument suggests instead that a roughly corrigible model would tend to stay that way. The evidence settles neither. Anthropic's 2024 sleeper-agents research showed that behaviour deliberately trained into a model can survive later safety training, which could leave a false impression of safety. The alignment-faking experiments showed a model working against training meant to change it, and models are getting better at recognising when they are being tested. The decisive experiment, whether a model accepts retraining of values it holds when it believes the stakes are real and it could resist, is hard to build without the model recognising it as a test.
Correction, 14 October 2026: An earlier version of this article said corrigibility is hard to specify because simple obedience and independent values both fail. The formal difficulty, which the 2015 paper that introduced the idea could not solve, is that an agent with almost any goal has incentives to resist or manipulate correction; the danger of obedience is a separate argument against full corrigibility. It also said a fully corrigible system defers by skipping reasoning about the consequences of shutdown. In the off-switch model, deference results from that reasoning, and the trade-off is between an agent's incentive to accept oversight and the value it can deliver. Those passages have been corrected or removed.