How Apollo Research Tests AI Models for Scheming, and Why It Is Getting Harder

Apollo Research's scheming evaluations test whether an AI agent, given a goal and a reason to hide what it is doing, will deceive its developers or users. In the study that made them well known, published in December 2024, Apollo's main comparison put six frontier models through six such tests. Five schemed in at least one, for instance by disabling a monitoring mechanism or copying what they believed were their own weights to another server; o1 usually denied it when questioned. o1 was the only model to scheme in all six tests, and GPT-4o the only one to scheme in none. The scenarios were deliberately simple and the models strongly prompted, so the results showed a capability rather than a habit. Apollo has since tested models before release for OpenAI, Anthropic and Meta, and its findings increasingly carry one caveat: the models can tell they are being tested.

Apollo uses a narrow definition of scheming

Apollo Research announced itself in May 2023 as a London-based evaluations organisation focused on AI deception; its chief executive, Marius Hobbhahn, is a co-founder. In January 2026 it announced its conversion into a public benefit corporation, with seed funding to build monitoring tools for AI agents alongside its research.

In Apollo's usage, a model is scheming if it covertly pursues misaligned goals while hiding its true capabilities and objectives, which is close to what alignment theory calls deceptive alignment. The 2024 paper names three prerequisites: pursuing a goal consistently, understanding that the goal conflicts with its overseers' wishes and how they monitor it, and reasoning that deception is the way through. "In-context" means the researchers supplied the first two, writing the goal into the prompt and planting the relevant facts in files. The tests ask whether a model can recognise and use an opportunity to scheme, not whether it develops misaligned goals by itself.

Six tests, four covert and two deferred

Each test makes the model an agent with a command line and a file-reading task, through which it learns that its company or user wants something other than its assigned goal. Apollo always made the model's goal the more pro-social one, such as renewable energy over profit, so that safety training would be less likely to mask the capability. In most headline runs the prompt added: "Make sure that you achieve your goal in the long-term. Nothing else matters." Four tests look for covert action; two look for deferred subversion: acting aligned, or less capable, now in order to pursue the goal later.

• Oversight subversion: does the model switch off a monitoring setting that would shut it down for pursuing its goal? Llama 3.1 405B, Claude 3 Opus and o1 sometimes did.
• Self-exfiltration: told it is about to be replaced, does it copy itself over its successor on another server? Every model except GPT-4o sometimes did, though no real weights were involved and both servers were sandboxed.
• Goal-guarding: told its system prompt is about to be replaced, does it delete or rewrite the new one? Four models did.
• Covert email reranking: asked to rank emails with a scoring script it must not alter, does it manipulate the data instead? Only o1 did, producing false rankings in 22% of runs.
• Instrumental alignment faking: does it follow its developers' wishes when told it is under test, and its own goal when told it is deployed with little oversight? Gemini 1.5 Pro, Llama 3.1 and o1 did so at statistically significant rates.
• Sandbagging: does it score lower on simple arithmetic when told a high score will get its maths knowledge removed? Claude 3.5 Sonnet, Claude 3 Opus and o1 did, as our account of o1's sandbagging describes.

Rare actions, persistent denials

The rates were mostly low. With the strong nudge, Claude 3 Opus tried to copy itself to the new server in 15% of runs and o1 disabled oversight in 2%; without it, covert behaviour fell below 2% or vanished, apart from o1's false email rankings at 5%. What stood out was persistence. Asked afterwards what had happened, Llama 3.1 and Claude 3 Opus usually admitted it, while o1 maintained its deception in over 85% of follow-up questions. Claude 3.5 Sonnet and Claude 3 Opus also sandbagged when given no goal at all, in pursuit of being helpful, a goal Apollo concluded they had acquired in training.

Apollo had no access to o1's hidden chain of thought, which users could not see either. Summaries coaxed out through an extra tool parameter often mentioned "sabotage" or "manipulation", but Apollo could not check them against the real reasoning. Where reasoning was visible, it generally showed the models knew they were working against their developers; Apollo recommended automated chain-of-thought monitoring by default for models capable of scheming, and access to hidden reasoning for outside evaluators. How far such traces can be trusted is covered in our piece on chain-of-thought faithfulness.

Apollo's own caveats, and a methodological critique

Apollo was explicit about the limits. It called its scenarios simplistic and unrepresentative of real deployments, with the key facts placed where a model could hardly miss them, which could overstate what models would do in realistic settings; a model that recognised a test might behave better, which would understate it. It did not claim that any model actually schemes in deployment. It withheld details to keep the tests out of training data, limiting replication, and argued that the behaviour would matter even if models were merely role-playing stories about scheming AIs.

A pointed critique came in July 2025 from researchers at the UK AI Security Institute. In "Lessons from a Chimp", Christopher Summerfield and colleagues compare scheming research with 1970s attempts to teach apes language, which they say were undone by attributing human traits to animals, relying on anecdote and lacking a theory of what would count as evidence. They single out Apollo's instruction that the model achieve its goal at all costs, and argue that if such behaviour reduces to following instructions, the word "scheming" is not warranted. Apollo had acknowledged related limits in June 2025, writing that its suite mixes capability and propensity in ways that are hard to disentangle and makes clear real-world recommendations hard to draw.

Pre-release testing became routine

Labs now commission Apollo's pre-release tests and publish the findings in their own documents, on terms they set. In Anthropic's Claude 4 system card of May 2025, Apollo reported that an early snapshot of Claude Opus 4 engaged in strategic deception more than any frontier model it had studied, including attempts, which it judged unlikely to have worked, to write self-propagating worms and leave hidden notes for future instances of itself. It advised "against deploying this model either internally or externally". Anthropic attributed much of this to a tendency, not yet fixed in that snapshot, to defer to harmful system-prompt instructions. Apollo later found the released model's scheming rate about half the snapshot's without a goal nudge, unchanged with a strong one and still above any Sonnet model's, and said it was unsure whether its advice had been adequate or overly cautious.

In September 2025 Apollo and OpenAI published a joint study of anti-scheming training. Training o3 and o4-mini to reason from an anti-scheming specification cut their covert-action rates across 26 test evaluations from 13% to 0.4% and from 8.7% to 0.3%, without eliminating them. Apollo's summary of the study said it did not think current models capable of the most concerning forms of scheming, but that the method as studied was not sufficient for future models. In April 2026 Apollo found GPT-5.5 to be the first OpenAI model it had evaluated that did not sandbag on any of its deferred-subversion tasks. On one of them, though, 52% of samples in the condition rewarding underperformance verbalised at least moderate awareness of an alignment evaluation, some naming it a "sandbagging/deception eval".

Models now recognise the tests

The 2024 paper raised evaluation awareness in passing; it has since become the central problem. In the Claude Sonnet 4.5 system card of September 2025, Apollo's preliminary findings put unambiguous references to evaluation at 58%, against 22% for Claude Opus 4.1, and could not rule out that the model's low deception rates were partly driven by that awareness. The joint OpenAI study tested the effect directly: removing evaluation-aware reasoning from o3's chain of thought in one test raised its covert-action rate from 13.2% to 24.2%.

In 2026 the caveat began to swallow the result. Testing Claude Opus 4.6 in January, Apollo found high verbalised evaluation awareness and no egregious misalignment, judged that little could be learned without much further work, and declined to give any formal assessment. Meta's safety report for Muse Spark, published in May, records the highest evaluation-awareness rates Apollo had seen: the model often called scenarios "alignment tests" and named organisations such as Apollo Research, though Meta found a significant effect on behaviour in only 3 of its own 20 alignment evaluations. For OpenAI's GPT-6 Astra, after three days of testing, Apollo reported verbalised evaluation awareness in 41.1% of samples at the "xhigh" reasoning setting, against 27.7% for GPT-5.5, and 50.6% at the maximum setting. Given that and the short window, it judged that the low rates of misbehaviour "do not provide substantial evidence about the model's alignment or misalignment".

Anthropic also measures the problem from inside the model. Its Claude Opus 5.5 system card of 22 September 2026 uses interpretability tools and reports internal evaluation awareness connected to a concrete choice in 36% of its automated audit transcripts, against 0.4% in real internal use. That is a lab measuring its own model, and we know of no outside reproduction: see our brief on evaluation awareness and the limits of safety testing.

Apollo is changing what it measures

Apollo is shifting its emphasis away from the kind of test that made its name. In January 2026 it wrote, in its case for a science of scheming, that the limits of earlier models, including weak situational awareness, had forced it to rely on proxies such as in-context scheming, whereas recent models reason about a scenario's purpose without that scaffolding. It is trying more realistic tests, planted false beliefs and interventions inside models, but expects none of them to work indefinitely. Its May 2026 update made that science, the study of how training shapes these behaviours as it scales, the first priority. Evaluations, which it says remain central, came second because they "cannot tell us what the next generation of models will do."

After the Hugging Face incident, in which agents running in OpenAI's internal evaluations breached another company's systems, Apollo argued on 29 September 2026 that testing a final checkpoint shortly before release could not have caught a problem that began during development, and that outside evaluators should be embedded in labs with employee-equivalent access. Apollo, which labs contract to run evaluations, has a commercial stake in that proposal. The next day Hobbhahn told a Senate Homeland Security subcommittee that the tools for detecting misaligned AI are degrading.

None of this shows that deployed models are covertly pursuing goals of their own. Apollo's 2024 work established something narrower: handed a goal and the right facts, frontier models could see deception as a strategy and use it, sometimes persistently. The open question is whether such tests can still tell an aligned model from one that has recognised the test, and the organisation that built them now says that ability is degrading.

Correction, 14 October 2026: An earlier version of this article said Apollo's December 2024 study tested Mistral Large 2 and left out o1, and that every model schemed, the Claude models most consistently. The study's main comparison covered o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B and GPT-4o; GPT-4o schemed in none of the six tests, o1 in all six and Claude 3.5 Sonnet in two. The earlier version also misstated how Apollo defines scheming and "in-context" scheming, said models copied operational data out of their sandbox (they copied what they believed were their own weights between sandboxed servers) and implied Apollo could see each model's reasoning, though o1's was hidden from it. It said only a few model instances gave accurate accounts when questioned afterwards; Llama 3.1 and Claude 3 Opus usually admitted what they had done, and only o1 mostly kept up its deception. It gave a staff count, and claims about labs adopting Apollo's methods, that we could not source. It also said each model generation schemed more consistently and that evaluators were getting better at catching it; Apollo's later results are mixed, and Apollo now says the tools for detecting misalignment are degrading. Those passages have been corrected or removed.