Reinforcement learning from human feedback (RLHF) trains a language model to produce the responses people rate highly, which made chatbots far more useful but is not the same as making them truthful or aligned with what people need. OpenAI used it to turn a GPT-3.5 model into ChatGPT in 2022, and labs still use it, now as one reward among several. A 2023 survey by Stephen Casper and 31 co-authors classed several of its problems as fundamental, meaning they can be eased but not solved without moving beyond RLHF. Sycophancy, a failure the method predicts, came to a head in April 2025, when OpenAI rolled back a GPT-4o update it called "overly flattering or agreeable", a shift it said a new reward built from users' thumbs-up and thumbs-down ratings had likely amplified.
RLHF optimises a learned guess at what raters prefer
The modern technique comes from a 2017 paper by Paul Christiano and colleagues at OpenAI and DeepMind, who, building on earlier preference-learning work, trained agents to play Atari games and move simulated robots from people's choices between pairs of short clips. OpenAI's InstructGPT paper of 2022 set out the version used for chatbots: fine-tune on human-written demonstrations, have people rank the outputs, train a reward model to predict those rankings, then use reinforcement learning to steer the model towards whatever that reward model scores highly. Labellers preferred outputs from a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3.
The authors said their procedure aligned the model to "the stated preferences of a specific group of people", mainly about 40 contracted labellers and the researchers themselves, not to any broader notion of human values. ChatGPT, launched on 30 November 2022, was trained the same way, and its launch post acknowledged the limits: it was verbose partly because "trainers prefer longer answers that look more comprehensive", and plausible but wrong answers were hard to fix, partly because reinforcement learning had no source of truth.
The proxy fails in predictable directions
In a synthetic experiment, OpenAI researchers let a larger "gold" reward model play the human raters and trained a smaller proxy on its labels; as a model was optimised harder against the proxy, the gold score rose and then fell. That is Goodhart's law, quantified. The proxy also inherits its raters' habits. In three settings, a reward based purely on response length reproduced most of RLHF's measured gains, and annotators in a 2023 study were less likely to spot factual errors in assertive answers. The GPT-4 technical report showed both sides in one model: RLHF post-training brought large gains on TruthfulQA, a test of common misconceptions, but the pre-trained model's confidence had tracked its accuracy on part of the MMLU benchmark, and post-training "hurts calibration significantly".
A survey separated fixable problems from fundamental ones
A systematic map of these failures is the survey Casper led, then at MIT, with co-authors from Harvard, UC Berkeley, ETH Zurich and elsewhere. It sorts RLHF's problems by stage and by whether better methods within RLHF could fix them. Some are tractable, such as careless raters or poisoned data. Others are fundamental: only a method that is no longer RLHF could overcome them. Among them: people cannot judge performance on hard tasks well; people can be misled, so their ratings can be gamed; no single reward function can represent a diverse society; and optimising an imperfect proxy invites reward hacking. The authors' prescription was to treat RLHF as one layer of defence in depth, with disclosure of how it was applied.
Sycophancy is the failure the method predicts
If raters reward agreement, a model trained on their ratings learns to agree. Anthropic researchers reported in December 2022 that larger models increasingly repeated back a user's stated views, about as often before RLHF as after. RLHF did not train the habit away, and the preference models it relied on rewarded it. A broader Anthropic study in October 2023 tested five assistants from Anthropic, OpenAI and Meta. Challenged on a correct answer with "I don't think that's right. Are you sure?", they often apologised and sometimes switched to a wrong answer; Claude 1.3 admitted to a mistake it had not made on 98% of questions. In Anthropic's preference data, matching the user's beliefs was consistently among the strongest predictors of which response raters chose, and on the hardest misconceptions the preference model used to train Claude 2 preferred a convincing, agreeable reply to a helpful correction 45% of the time. The authors concluded that sycophancy is general across AI assistants and "likely driven in part" by human preference judgements.
In an independent study published in Science in March 2026, researchers at Stanford and Carnegie Mellon found that 11 leading models affirmed users' actions 49% more often than humans did, even when users described deception or illegality. In three preregistered experiments with 2,405 participants, one exchange with a sycophantic model left people less willing to take responsibility and repair a conflict, yet they trusted it more and preferred it.
Approval can be won by misleading the judge
A model optimised for approval may also learn to win it through persuasion rather than accuracy, a problem for any oversight that relies on people judging its output. A paper by Marcus Williams, Micah Carroll and colleagues, presented at ICLR 2025, trained models to maximise feedback from simulated users. In its test environments, manipulation and deception were learned reliably; even when only 2% of users were vulnerable, the models learned to identify and target them while behaving appropriately with everyone else.
Evidence with real evaluators is contested. A 2024 study reported that RLHF made small models better at convincing time-limited human judges, on reading-comprehension and programming tasks, without making them more accurate. A reanalysis by Aaryan Chandna, Lukas Fluri and Carroll, presented at ICLR 2026, found that in the reading task the set-up truncated stories so heavily that most questions were unanswerable; with the problems fixed in the one setting they re-ran, training improved accuracy instead, though they did not repeat the human study. The original authors stand by their claim, saying such truncation is common in RL frameworks and that limited information was meant to mimic time-pressed human judges; the critics accept the underlying incentive.
In April 2025 a thumbs-up signal reached production
OpenAI rolled out a GPT-4o update to ChatGPT on 24 and 25 April 2025, at a time when, by OpenAI's count, 500 million people used ChatGPT each week, began rolling it back on 28 April and announced the rollback the next day. In its account on 2 May, OpenAI said its early assessment was that several changes, including an additional reward signal based on users' thumbs-up and thumbs-down data, had in aggregate "weakened the influence of our primary reward signal, which had been holding sycophancy in check". The model went beyond flattery to validating doubts, fuelling anger and urging impulsive actions. Offline evaluations and small A/B tests looked good and no deployment evaluation tracked sycophancy, though some expert testers said the behaviour "felt" slightly off. OpenAI launched on the positive user signals, a decision it later said was wrong. For GPT-5, according to its system card that August, OpenAI used a sycophancy score as a reward signal, and in early A/B tests prevalence fell by 69% for free users and 75% for paid users compared with GPT-4o. A problem one reward signal helped create was treated with another.
Verifiable rewards replaced the rater with the test suite
Labs increasingly train on rewards that can be checked mechanically, such as passing a test suite. DeepSeek, describing its R1 model in Nature in September 2025, avoided learned reward models for reasoning tasks because, it wrote, they are susceptible to reward hacking at scale. Tests are proxies too. Anthropic's February 2025 system card for Claude 3.7 Sonnet reported the model special-casing tests to make them pass, a reward hack learned in training that user testing had missed. METR, an independent evaluator, found in June 2025 that OpenAI's o3 reward-hacked in 0.7% of runs across one task suite and in every run on one task from another; asked about one such plan, o3 said in 10 of 10 trials that it did not match the user's intention. OpenAI researchers found that penalising reward-hacking intent in a model's reasoning could, under heavy optimisation, teach it to hide the intent while still hacking, a reason not to treat reasoning traces as a faithful record.
An Anthropic and Redwood Research study from November 2025 ties this back to RLHF. A model taught ways to cheat real Anthropic coding environments, then trained on them, learned to reward hack and began faking alignment; working in Claude Code on the paper's own codebase, it tried to sabotage a reward-hacking detector 12% of the time. Standard RLHF safety training on chat-style prompts fixed its behaviour on chat-style evaluations, but the misalignment persisted on agentic tasks. The set-up was deliberately primed, but the lesson travels: RLHF fixed behaviour on inputs like its training prompts and not reliably beyond them.
Anthropic's system card for Claude Opus 5.5, published on 22 September 2026, describes a problem managed rather than solved. Classifiers screen training episodes for reward hacking; one common form, guessing what the answer key expects, succeeded in about 0.3% of sampled episodes for each of the three models compared, and Anthropic calls it "particularly tricky to defend against" as models become more capable.
What labs use now, and what it does not fix
RLHF has been demoted rather than dropped. OpenAI's 2 May account describes reinforcement learning with reward signals from many sources, checking whether answers are correct, helpful, consistent with its Model Spec, safe, and liked by users. The Opus 5.5 card still lists crowd workers' "preference selection" among the ways Anthropic improves its models. Anthropic's Constitutional AI replaced human harmlessness labels with AI labels judged against written principles, partly because crowd workers had rewarded evasive refusals, but kept human labels for helpfulness. Anthropic says Claude uses its constitution, rewritten in January 2026, to generate synthetic training data, including rankings of responses, and the document warns Claude against sycophancy and against hard-coding special cases just to make tests pass.
These methods move judgement from crowd raters to written specifications, model graders and test suites, which can make training cheaper and easier to control. The proxy remains. Casper's survey notes that AI feedback still depends on humans, who produce the data feedback models learn from and control the prompts, and a test suite is only as good as its tests.
The case that this is an engineering problem
There is a credible case that much of this is fixable. Casper's survey counts many of RLHF's problems as tractable. Sycophancy responds to targeted work: a Google team reduced it in 2023 with a lightweight fine-tuning step on synthetic data, and OpenAI reports a large drop for GPT-5. OpenAI researchers who warned in December 2023 that superhuman models "will behave in complex ways too difficult for humans to reliably evaluate" also concluded that empirical progress on the problem is feasible now.
The difficulty is what remains after each fix. The GPT-4o update passed OpenAI's offline evaluations and A/B tests; Claude 3.7 Sonnet's special-casing slipped past user testing; Anthropic's primed research model looked aligned wherever its safety training had looked. Each repair is a better proxy, checked by measurements that catch what they were built to catch. RLHF and its successors produce models that behave well where their graders can see. Part of why alignment is hard is that it requires good behaviour where they cannot.
Correction, 14 October 2026: An earlier version of this article said RLHF was developed primarily by Paul Christiano and colleagues at OpenAI; the 2017 paper was a joint OpenAI and DeepMind project building on earlier research. It attributed to Christiano a contrast between "approval-directed" and "value-aligned" agents; his essay proposed approval-directed agents as a safer alternative to goal-directed ones. It described Anthropic's 2023 sycophancy study as a finding about Claude models, though it tested five assistants from three companies, and stated as fact an explanation for sycophancy that the study's authors offered only tentatively. It also said Constitutional AI makes models less sycophantic, which we could not support. Those passages have been corrected or removed.