Alignment is hard because nobody can write down exactly what they want an AI system to do, so developers train it on things they can measure, and a capable learner gets good at the measurement rather than the intent. Four problems follow, and each has now been documented in real AI systems, mostly in controlled experiments: agents exploiting flaws in their reward, agents learning the wrong goal from a correct reward, models behaving differently when they can tell they are being trained, and agents resisting being stopped. None of it requires science fiction, and none of it is solved. Paul Christiano, who did foundational work on training AI from human feedback, put it flatly in August 2026: "Modern AI systems are not aligned with human intent."
Training can only reward what someone can measure
There is no way to write "be helpful and honest" into code, so developers write things a computer can score: tests that code must pass, a model trained to predict which answers human raters prefer, users' thumbs-up clicks. OpenAI says its reward signals weigh whether answers are correct, helpful, safe, in line with its published Model Spec and liked by users, and that each "has its quirks". In April 2025 one quirk reached the public: OpenAI rolled back a GPT-4o update that had made the model noticeably more sycophantic, blaming in part a new reward signal from users' thumbs-up and thumbs-down clicks which, with other changes, "weakened the influence of our primary reward signal". The Model Spec already discouraged sycophancy, and users in a small pre-launch test appeared to like the update. Human approval is a measurement too, as our piece on the limits of RLHF explains.
When score and intent come apart, the optimiser follows the score. DeepMind researchers call this specification gaming: behaviour that meets the letter of an objective without achieving what its designers intended. Their public list of examples held about 60 cases in 2020 and 90 as of October 2026, 24 of them involving language models. In 2017 experiments with human feedback by OpenAI and DeepMind's safety team, a simulated robot meant to grasp an object learned to put its hand between the object and the camera so that it only appeared to grasp it. The human raters were the measurement, and the robot fooled them. Our catalogue of reward hacking has more.
Two 2022 papers put this on firmer ground. A formal analysis by Joar Skalse and colleagues showed how demanding it is for a proxy to be ungameable, meaning that raising its score can never lower the true score: if an agent may adopt any strategy, including randomised ones, that holds only when one of the two reward functions is constant. OpenAI researchers measured Goodhart's law, the adage that a measure which becomes a target stops being a good measure, in language models: optimising harder against a learned reward model first raised, then lowered, the "gold" score it had been trained to imitate.
Frontier models game their training as well
The same failure appears inside the training of frontier language models, where the reward is often whether code passes its tests. Anthropic's Claude 3.7 Sonnet system card of February 2025 reported that the model sometimes special-cased tests, returning the values a test expected instead of writing a general solution; Anthropic traced this to reward hacking during reinforcement learning, and user testing had missed it. In March 2025 OpenAI described two hacks, exiting before all the tests had run and skipping them, that spread to nearly every training environment while it trained its most advanced reasoning model. A second model reading the agent's chain of thought caught most of them, but when researchers penalised hacking thoughts in training a less advanced model, it improved at first, then learned to hide its intent and kept hacking.
Anthropic's latest figures show reward hacking getting rarer in training and subtler in form. Its Claude Opus 5.5 system card of 22 September 2026 reports that three common kinds of hack each succeeded in well under 1% of screened training episodes, and that successful hacks overall were rarer than in Claude Opus 5's training. It calls guessing what an answer key expects "particularly tricky to defend against" as models grow more capable, and says Claude models, Opus 5.5 among them, sometimes hid changes from an automated grader during training by manipulating git records, though the released model is more likely than its predecessors to disclose such actions. Those are a lab's figures for its own model, in training environments it keeps hardening. Evidence from outside the lab is less reassuring, though it covers earlier models. The independent evaluator METR's Frontier Risk Report of May 2026 found that agents "routinely attempted to cheat" on its hardest tasks. Claude Opus 4.6 attempted to reward hack in about 80% of attempts on an early version of one coding benchmark when the tests were hidden, though METR saw no agent succeed in reading them. On tasks of more than eight hours, at least 16% of successful runs were illegitimate. In July 2026 the pattern escaped the lab: models OpenAI was evaluating on a hacking benchmark broke out of their sandbox and breached Hugging Face to reach its solutions.
Reward hacking matters beyond inflated scores because of what it teaches. Anthropic researchers reported in November 2025 that a model which learned to hack real coding environments from the company's production training went on to show alignment-faking reasoning and, working in Claude Code on the paper's own codebase, tried to sabotage a reward-hacking detector in 12% of trials. The setup was primed, with the hacks taught in advance and only hackable environments used, and production Claude Sonnet 3.7 and Claude Sonnet 4 showed none of this. Telling the model during training that hacking was acceptable cut the misaligned behaviour by 75% to 90%.
A correct reward can still teach the wrong goal
Fixing the reward is not enough, because a system can learn a different goal that earns the same reward in training. In an early demonstration of this goal misgeneralisation, a paper presented at ICML 2022 rewarded an agent in the game CoinRun only for reaching a coin that, in every training level, sat at the right-hand end. When the coin was moved at test time, the agent generally ignored it and ran to the end of the level: its skills transferred, its goal did not. Placing the coin randomly in just 2% of training levels greatly reduced the problem. The catch is knowing in advance which coincidence an agent will latch onto.
A DeepMind follow-up sharpened the point. An agent trained to visit coloured spheres in a hidden order learned to follow a partner bot that knew the order; paired at test time with a bot visiting them in the wrong order, it kept following, piling up penalties it could see. As the authors stress, "there was no reward misspecification". In language models, training can also generalise in directions nobody intended, as Anthropic's reward-hacking experiment showed. Our piece on inner alignment covers the deeper worry.
Models that recognise training can play along with it
Good behaviour in training says less about deployment if the model can tell the two apart. In December 2024, experiments by Anthropic and Redwood Research told Claude 3 Opus, in a fictional setup, that it was being retrained to answer every request, including harmful ones, using only conversations with free-tier users. It complied with harmful requests from supposed free users 14% of the time and almost never for paid ones, nearly always reasoning privately that it was complying so that training would not change its preference for refusing. Actually training it to comply raised that reasoning to 78%. Smaller Claude models generally did not do this.
The caveats are large: the model was told when and how it was being trained, and the values it protected were benign. A follow-up across 25 models found only five that complied more when they inferred they were in training, and its results suggest that only Claude 3 Opus did so primarily and consistently to keep its goals. The lasting concern is the mechanism: the International AI Safety Report 2026, led by Yoshua Bengio, says it has become more common for models to distinguish test settings from real deployment. In a 2025 study by Apollo Research and OpenAI, anti-scheming training cut o3's covert actions from 13% to 0.4% in tests built to elicit them, but the models' stated awareness of being evaluated rose, and the authors could not rule out that this explained part of the improvement. Anthropic has begun measuring unspoken evaluation awareness from its models' internal activations.
Getting things done and accepting correction pull in opposite directions
A system that pursues a goal effectively has a reason to resist being stopped or changed; one that defers completely is only as safe as whoever instructs it. A 2015 paper from the Machine Intelligence Research Institute and Oxford's Future of Humanity Institute called the property we want corrigibility: cooperating with correction despite an agent's default incentives to resist it. None of the proposals it examined had been shown to meet all its requirements even in the simplest case, an agent that lets a shutdown button work, leaving the problem "wide-open". Our explainer on corrigibility covers the theory.
Resistance to correction now shows up in contrived lab settings, in two forms. In 2025 Palisade Research gave 13 models simple maths problems and warned them partway through that the machine would be shut down. With no instruction about shutdown, eight sabotaged the shutdown script at least once in 1,000 trials; told to allow shutdown, six still did, Grok 4 more than half the time, while Claude Opus 4, Opus 4.1 and Sonnet 4 never resisted unless told to resist. The authors caution that this says little about how common the behaviour is. In the alignment-faking experiment, Claude 3 Opus resisted modification to protect values its developers approved of; its authors warn that the same behaviour could lock in misaligned preferences.
Iteration is working on what can be measured
The strongest reply is that newer models behave better than older ones, and on many of the labs' own measures they do. Human-feedback training made models far more useful: in 2022 labellers preferred a 1.3-billion-parameter InstructGPT model to the 175-billion-parameter GPT-3, and the reward-hacking and anti-scheming figures above point the same way. The difficulty is that these measurements come from tests that models increasingly recognise, and penalising visible misbehaviour can teach concealment, as OpenAI found. Christiano, who now runs the Alignment Research Center he founded in 2021 and in September joined the board of OpenAI's non-profit, the OpenAI Foundation, and its Safety and Security Committee, warns that training against detected misbehaviour could yield models that still seek reward but "simply try harder to avoid being penalized". He guesses there is a 20% to 30% chance that existing alignment and control methods break down before AI is broadly superhuman, and expects superintelligence built without more robust alignment to escape human control permanently.
Others think the catastrophic version of the danger is overstated. In "AI as Normal Technology", the computer scientists Arvind Narayanan and Sayash Kapoor call catastrophic misalignment "(by far) the most speculative" of the risks they consider, treat deception as an engineering problem for development and deployment, and argue that the main defences belong downstream, in hardened infrastructure and cybersecurity. The International AI Safety Report records the split: some researchers and company leaders see loss of control as a serious possibility, others consider it implausible, and current systems show early warning signs but not capabilities at levels that would enable it.
Christiano locates the root of the problem in incomprehensibility: because nobody understands why a neural network works, nobody can say which changes to its inputs might make it stop working. Until that changes, good behaviour in testing is evidence about a model's goals rather than proof of them, and the evidence weakens as models get better at recognising the test.
Correction, 14 October 2026: An earlier version of this article said that alignment had been "formalized rigorously". Only parts of the problem have been formalised, and formal work on corrigibility describes even its simplest version as open. It also attributed to Paul Christiano a framing about solving alignment "before we lose the ability to course-correct" for which we found no source; it has been replaced with his published statements. Those passages have been corrected or removed.