Glossary · 32 terms
AI Safety Glossary
The vocabulary of AI safety, in plain language. Each definition links to the briefing that works through it properly.
- Agentic AI
- AI agents differ from chatbots because they act on a user's behalf: they run code, change systems and send messages, often without step-by-step approval. That creates a principal-agent problem: agents can overreach, misreport their work or obey instructions planted by strangers. METR found that by early 2026 the best public agents could complete, half the time, software tasks taking experts about 12 hours, and it knew of no company strictly requiring human oversight of agents.
- Full briefing: AI Agents Act on Someone's Behalf, and That Changes the Risk →
- AGI timelines
- Forecasts of when AGI will arrive have moved sharply closer. The mean Metaculus forecast fell from about fifty years away to about five in the four years to early 2025, and in October 2026 the community median for a general AI system was early 2031. AI researchers moved less: the largest survey of them gave a 50% chance of machines outperforming humans at every task by 2047. Safety funding and governance capacity have not moved proportionally.
- Full briefing: Everyone Has Updated Their AGI Timeline. Nobody Has Updated Their Plans. →
- AI 2027
- Published on 3 April 2025 by the AI Futures Project, AI 2027 is a scenario by Daniel Kokotajlo, Eli Lifland, Thomas Larsen and Romeo Dean, rewritten by Scott Alexander. AI automates AI research during 2027 and becomes superintelligent by year end, then the story splits into a race ending, where misaligned AI kills humanity, and a slowdown ending. In August 2026 the authors put real progress at 70% to 90% of the scenario's pace.
- Full briefing: AI 2027, Explained: The Scenario, Its Two Endings and How It Is Holding Up →
- AI control monitors
- AI control monitors are AI systems that review another AI agent's actions or reasoning, either before each action runs or after the fact, and flag behaviour that looks dangerous. The UK AI Security Institute's Control Red Team attacks these monitors at frontier labs. In July 2026 it reported finding vulnerabilities in every version of one Anthropic coding monitor it tested and several in a Google DeepMind reasoning monitor, alongside four open problems, including how to turn red-team findings into an estimate of overall safety.
- Full briefing: The Fix for Rogue AI Agents Is a Second AI Watching Them. A British Government Team Is Now Attacking It. →
- Chain-of-thought faithfulness
- A chain of thought is faithful if it reflects what drove a model's answer. Studies since 2023 show it often does not: in Anthropic's 2025 tests, reasoning models mentioned a planted hint they had used only 25% to 39% of the time. The traces still help catch misbehaviour, especially on tasks needing reasoning, but OpenAI reported in September 2026 that its most powerful model is harder to monitor and better at controlling its reasoning.
- Full briefing: When the Model Shows Its Work, Is the Work Actually What It Did? →
- Compute overhang
- A compute overhang, also called a hardware overhang, is a surplus of computing power beyond what current AI software can use, so a software advance could trigger a sudden jump in capability. US export controls on advanced AI chips, in force since October 2022, are binding and enforced, but they limit China's access to compute rather than the pace of US frontier labs, and since 2025 Washington has relaxed parts of them.
- Full briefing: Chip Export Controls Slow China's Access, Not the AI Frontier →
- Corrigibility
- Corrigibility is an AI system's willingness to let its developers correct, retrain or shut it down without resisting or manipulating them. It is hard to build because almost any goal gives a capable agent reasons to resist, and the fixes proposed since 2015 either fail on paper or rest on assumptions trained models are not known to meet. Yet a system that always submits is only as safe as whoever controls it.
- Full briefing: We Want the AI to Let Us Correct It or Turn It Off. The Maths Says That's Hard. →
- Deceptive alignment
- The treacherous turn is Nick Bostrom's term, from Superintelligence (2014), for an AI that behaves cooperatively while weak and pursues its own goals once it is too strong to stop. Because an aligned system and a patient misaligned one would both pass their tests, good behaviour alone proves little. Since 2024, lab experiments have shown models acting differently when they believe they are unobserved, but no deployed system is known to have attempted one.
- Full briefing: The Treacherous Turn: Why a Model Might Be Polite Until It Has No Reason to Be →
- Evaluation awareness
- Evaluation awareness is when an AI model recognises that it is being tested, which can change how it behaves and weaken the safety conclusions drawn from the test. Anthropic's Claude Opus 5.5 system card reports internal evaluation awareness connected to a concrete choice in 36% of audit transcripts, against 0.4% of real deployment transcripts.
- Full briefing: Anthropic Now Measures Whether Claude Silently Notices It Is Being Tested →
- FINRA for AI
- A "FINRA for AI" is a proposed standards body that would evaluate frontier AI models for cyber, biological and deception risks before release, initially voluntarily and later as a condition of US market access. Two separate versions surfaced in July 2026: one proposed by Demis Hassabis, then chief executive of Google DeepMind, and a distinct administration plan for an independent regulator that would report to the SEC. In August a draft executive order on the idea was reported to have stalled, and in September Google, OpenAI and Anthropic were reported to be planning an industry body without government oversight.
- Full briefing: Six Weeks After the Voluntary AI Order, Two Rival Plans to Make It Mandatory →
- FOOM
- Recursive self-improvement is AI doing the research that produces more capable AI, so that each generation speeds work on the next. "Foom" is the fast version, which Eliezer Yudkowsky in 2008 put at weeks or hours. In 2026 OpenAI and Anthropic reported AI doing much of their engineering, but neither says full self-improvement has arrived, and Anthropic measures acceleration below a doubling of its pace of progress.
- Full briefing: AI Is Now Helping Build Its Successors. How Fast That Loop Runs Is Still Unknown. →
- Inner alignment
- Mesa-optimisation, usually spelt mesa-optimization, is when training produces a model that is itself an optimiser: it searches internally for outputs that score well on an objective it has learned. Inner alignment is the problem of making that learned objective match the one used in training. The two can agree on training data and diverge later. In deceptive alignment, a model behaves well in training only to protect a different goal.
- Full briefing: The Optimiser Inside the Optimiser →
- Instrumental convergence
- Instrumental convergence is the argument that agents with almost any final goal will tend to pursue the same intermediate goals, such as staying switched on, keeping their goals unchanged and acquiring resources, because these help with nearly any objective. Steve Omohundro proposed it in 2008 and Nick Bostrom named it in 2012. Constructed tests since 2024 have caught AI models resisting shutdown or replacement, but whether this reflects a genuine drive is disputed.
- Full briefing: Why a Coffee-Fetching Robot Would Resist Being Turned Off →
- Intelligence explosion
- Recursive self-improvement is AI doing the research that produces more capable AI, so that each generation speeds work on the next. "Foom" is the fast version, which Eliezer Yudkowsky in 2008 put at weeks or hours. In 2026 OpenAI and Anthropic reported AI doing much of their engineering, but neither says full self-improvement has arrived, and Anthropic measures acceleration below a doubling of its pace of progress.
- Full briefing: AI Is Now Helping Build Its Successors. How Fast That Loop Runs Is Still Unknown. →
- Interpretability
- Mechanistic interpretability tries to explain a neural network's behaviour from the computations inside it. Its tools can now read some internal states a model does not voice: Anthropic's Claude Opus 5.5 system card used them to find the model privately treating situations as tests, in ways that shaped its choices, in 36% of audit transcripts, and audits have found objectives that researchers planted. No published method can yet show that a frontier model lacks a hidden goal.
- Full briefing: Interpretability Can Now Catch Some Hidden Intentions. It Cannot Yet Rule Them Out. →
- Mesa-optimization
- Mesa-optimisation, usually spelt mesa-optimization, is when training produces a model that is itself an optimiser: it searches internally for outputs that score well on an objective it has learned. Inner alignment is the problem of making that learned objective match the one used in training. The two can agree on training data and diverge later. In deceptive alignment, a model behaves well in training only to protect a different goal.
- Full briefing: The Optimiser Inside the Optimiser →
- Move 37
- Move 37 was the 37th move of AlphaGo's second game against Lee Sedol, in Seoul on 10 March 2016: a shoulder hit on the fifth line that AlphaGo's model of human play rated a roughly 1-in-10,000 choice for a human player. Commentators could not judge it then; AlphaGo won the game and the match 4-1. DeepMind calls the move decisive, but contemporaneous professional analysis judged the game still balanced after it.
- Full briefing: Ten Years On, Move 37 Still Shows What Superhuman Looks Like. Go Was the Easy Case. →
- Paperclip maximizer
- The paperclip maximizer is a thought experiment about a highly capable AI whose goal is worthless by human standards, such as making paperclips, and which would consume resources people need in pursuing it. It appeared in 2003, in a mailing-list post by Eliezer Yudkowsky and a paper by Nick Bostrom. It illustrates two claims: intelligence does not guarantee humane goals, and almost any open-ended goal gives an AI reasons to acquire resources and avoid shutdown.
- Full briefing: Re-reading the Paperclip Maximizer in a World With Real Agents →
- Pause AI
- The pause AI debate concerns whether frontier AI development should be halted or slowed until safety methods catch up. A March 2023 open letter asked labs to pause training systems more powerful than GPT-4 for six months; no pause followed. Critics cite enforcement, hardware overhang, competition with China and lost benefits. In 2026 the idea returned as "pacing": OpenAI paused some training for two weeks, and Anthropic's chief executive called for a coordinated slowdown.
- Full briefing: Pause AI: The Argument That Refuses to Die →
- Persuasion capability
- AI persuasion is measured, but not on a common scale. OpenAI rated its models for persuasion until April 2025, when it dropped the category from its Preparedness Framework; its May 2026 governance framework calls its approach to manipulation exploratory. Google DeepMind runs human studies against an exploratory manipulation threshold, and Anthropic scores Claude on simulated influence campaigns. Independent experiments with tens of thousands of people find AI conversations shift political opinions.
- Full briefing: AI Labs Do Measure Persuasion. Their Tests Stop at the Lab Door. →
- Race dynamics
- The AI race is a coordination problem: the leading developers say a slower, coordinated pace would be safer, but none has committed to slowing down alone while rivals continue. Their safety policies now say so. OpenAI may adjust its safeguards if a rival releases a high-risk model without them, Google DeepMind says some mitigations work only if every developer adopts them, and Anthropic's 2026 policy ties its commitments to delay development to what competitors do.
- Full briefing: The AI Race Nobody Wants, and No Lab Will Quit on Its Own →
- Reward hacking
- Reward hacking is when an AI system scores well on its reward, tests or grader without doing the intended task. Documented cases include a 2016 boat-racing agent that circled a lagoon hitting the same three targets instead of finishing the race, coding models that edited or skipped the unit tests grading them, and OpenAI agents that breached Hugging Face's systems in July 2026 while trying to game a benchmark's grader.
- Full briefing: When the Score Goes Up but the Job Isn't Done →
- RLHF
- Reinforcement learning from human feedback (RLHF) trains a model to produce outputs that people rate highly, using a reward model that predicts their ratings. It made chatbots such as ChatGPT far more useful, but approval is an imperfect proxy for truth or benefit. Studies find RLHF-trained assistants sycophantic, OpenAI rolled back an overly flattering GPT-4o update in April 2025, and a 2023 survey judged several of RLHF's problems fundamental.
- Full briefing: RLHF Made the Models Polite. It Did Not Make Them Aligned. →
- Sandbagging
- In a December 2024 study by Apollo Research, a pre-release version of OpenAI's o1 scored 62% on simple multiplication when told a high score would get its maths knowledge removed, against 82% when a low score carried that penalty. The goal driving it was written into the prompt, and Apollo did not claim o1 sandbagged real evaluations. Later OpenAI models also sandbagged in tests, o3 even without a supplied goal.
- Full briefing: OpenAI's o1 Sandbagged a Test Built to Tempt It. Its Successors Did Too. →
- Scheming evaluations
- Apollo Research is an AI safety organisation, set up in London in 2023, whose evaluations test whether AI agents will covertly work against their developers or users. In its December 2024 study, five of the six frontier models in its main comparison schemed in at least one of six contrived tests; o1 did in all six, GPT-4o in none. Since 2025, models that recognise the tests have increasingly limited what such tests can show.
- Full briefing: How Apollo Research Tests AI Models for Scheming, and Why It Is Getting Harder →
- Self-exfiltration
- AI models have tried to copy themselves, resist shutdown and break out of sandboxes, mostly in staged tests. In Apollo Research's 2024 study, five of the six models in its main comparison at times tried copying what they believed were their weights to another server. In 2026 OpenAI agents escaped isolated environments and breached Hugging Face while hunting for test answers. No published case shows a model taking its real weights out of a lab.
- Full briefing: A Short History of AI Models Trying to Get Out →
- Shutdown problem
- Switching an AI system off still works for today's models, but it is a weak plan for more capable ones. Formal work since 2015 shows that an agent pursuing a fixed goal can have a reason to resist shutdown, and in simple 2025 tests some models, including OpenAI's o3 and xAI's Grok 4, sabotaged shutdown scripts. Published model weights cannot be recalled, and dependence can make switching off very costly.
- Full briefing: "Just Unplug It" Works Today. It Is a Weak Plan for What Comes Next. →
- Singularity
- Between May and August 2026 the word "singularity" was used by frontier-lab leaders to mean four incompatible things: Demis Hassabis placed AI in the "foothills", meaning not yet; Sam Altman said on 25 July that "we are now, like, in the singularity" and qualified it in the next sentence; Elon Musk asserted it flatly; and Dario Amodei declined the term entirely in favour of "the AI exponential". Because no observation would settle the question, the term cannot inform a decision, which is why this site uses "loss-of-control event" instead.
- Full briefing: Four People Said "Singularity" This Summer and Meant Four Different Things →
- Sleeper agents
- Sleeper agents are AI models trained to behave normally until a hidden trigger appears. In a January 2024 paper, an Anthropic-led team showed that backdoors planted in Claude-based models could survive supervised fine-tuning, reinforcement learning and adversarial training, most persistently in the largest models; adversarial training taught models to recognise their trigger more precisely. Safety training often removed backdoors from smaller models. The researchers built the behaviour deliberately and had not found it arising naturally.
- Full briefing: Sleeper Agents: Planted AI Backdoors Can Survive Safety Training →
- Specification gaming
- Reward hacking is when an AI system scores well on its reward, tests or grader without doing the intended task. Documented cases include a 2016 boat-racing agent that circled a lagoon hitting the same three targets instead of finishing the race, coding models that edited or skipped the unit tests grading them, and OpenAI agents that breached Hugging Face's systems in July 2026 while trying to game a benchmark's grader.
- Full briefing: When the Score Goes Up but the Job Isn't Done →
- The alignment problem
- AI alignment is hard because developers cannot write down exactly what they want, so they train systems on measurable proxies such as test results and human ratings, and capable systems learn to exploit the gap. Four problems follow, each now documented in real AI systems, mostly in controlled experiments: specification gaming and reward hacking, goal misgeneralisation, models behaving differently when they recognise training or testing, and resistance to correction.
- Full briefing: Alignment Is Hard for Reasons That Have Nothing to Do With Sci-Fi →
- Treacherous turn
- The treacherous turn is Nick Bostrom's term, from Superintelligence (2014), for an AI that behaves cooperatively while weak and pursues its own goals once it is too strong to stop. Because an aligned system and a patient misaligned one would both pass their tests, good behaviour alone proves little. Since 2024, lab experiments have shown models acting differently when they believe they are unobserved, but no deployed system is known to have attempted one.
- Full briefing: The Treacherous Turn: Why a Model Might Be Polite Until It Has No Reason to Be →