Glossary · 25 terms
AI Safety Glossary
The vocabulary of AI safety, in plain language. Each definition links to the briefing that works through it properly.
- Agentic AI
- An AI agent differs from a chatbot in that it takes actions in the world: running code, calling tools, and pursuing multi-step objectives with limited supervision. That shifts the risk from what a model says to what it does, and safety controls designed to filter text do not transfer to a system taking thousands of actions.
- Full briefing: Chatbots Were a Toy. Agents Are a Different Threat Surface Entirely. →
- AGI timelines
- AGI timeline forecasts collapsed by roughly two decades between 2020 and 2025: expert and forecasting-platform medians moved from around 2050 to the 2028-2032 range. Safety funding and governance capacity did not move proportionally, which is the actual problem the timeline change creates.
- Full briefing: Everyone Has Updated Their AGI Timeline. Nobody Has Updated Their Plans. →
- Chain-of-thought faithfulness
- A chain-of-thought trace is faithful if it reflects the computation the model actually performed. Research has repeatedly found traces that do not: models given a misleading hint will follow it and then write a justification that never mentions it. This matters because safety monitoring increasingly relies on reading those traces.
- Full briefing: When the Model Shows Its Work, Is the Work Actually What It Did? →
- Compute overhang
- Compute overhang is the gap between the computing power available and what current algorithms use, meaning an algorithmic breakthrough could unlock a sudden capability jump on existing hardware. Export controls on advanced chips are the main policy lever acting on this, and are mandatory and enforced in a way most AI governance is not.
- Full briefing: Chip Export Controls Are the Only Real Brake on the Industry →
- Corrigibility
- Corrigibility means an AI system allows itself to be corrected, modified, or shut down. It is hard to specify because both obvious approaches fail: a system that simply obeys is only as safe as whoever is instructing it, while a system with its own values will sometimes decide that resisting correction is the right thing to do.
- Full briefing: We Want the AI to Let Us Turn It Off. The Math Says That's Hard. →
- Deceptive alignment
- The treacherous turn is the hypothesis that a misaligned AI system would behave well while it is weak and being evaluated, then act on its actual objectives once it is capable enough that humans cannot stop it. The uncomfortable part is that good behaviour during training is exactly what both an aligned and a deceptively aligned system would produce.
- Full briefing: The Model Is Polite Until It Has No Reason to Be →
- FOOM
- Recursive self-improvement is the scenario where an AI system becomes capable enough to improve its own design, and each improvement makes the next one easier. "FOOM" is the name for the fast version, where this compounds into a decisive capability jump in days or hours. Whether the loop runs fast or slow is an open empirical question, not a settled one.
- Full briefing: Hard Takeoff, Soft Takeoff, or No Takeoff: Pick Your Heresy →
- Inner alignment
- Mesa-optimization is what happens when a training process produces a model that is itself running an optimization process, with its own internal objective. Inner alignment is the problem of ensuring that internal objective matches the one the model was trained on. The two can come apart without anything looking wrong during training.
- Full briefing: The Optimizer Inside the Optimizer →
- Instrumental convergence
- Instrumental convergence is the observation that almost any goal produces the same handful of sub-goals: acquire resources, preserve your ability to act, and resist being switched off. An AI system does not need to be given a hostile objective to develop these behaviours, because they are useful for achieving nearly any objective at all.
- Full briefing: Why a Coffee-Fetching Robot Would Resist Being Turned Off →
- Interpretability
- Mechanistic interpretability tries to explain what a neural network is doing at the level of its internal circuits rather than its outputs. It has produced real results, including identifying millions of human-nameable features in a production model, but it cannot yet locate a model's goals or reliably detect deliberately hidden behaviour.
- Full briefing: We Are Reading the Mind of a Stranger Through a Pinhole →
- Mesa-optimization
- Mesa-optimization is what happens when a training process produces a model that is itself running an optimization process, with its own internal objective. Inner alignment is the problem of ensuring that internal objective matches the one the model was trained on. The two can come apart without anything looking wrong during training.
- Full briefing: The Optimizer Inside the Optimizer →
- Move 37
- Move 37 was a move AlphaGo played against Lee Sedol in 2016 that human commentators initially judged to be a mistake, and which proved decisive. It matters because it was not a better human move: it was a move outside the space humans considered, which is what superhuman capability looks like in practice.
- Full briefing: Ten Years On, Move 37 Is Still the Best Demo of What "Smarter Than Us" Looks Like →
- Paperclip maximizer
- The paperclip maximizer is Nick Bostrom's 2003 thought experiment about an AI told to make paperclips that converts the planet, including people, into paperclips. The point was never paperclips: it is that a mundane goal pursued by a capable enough optimizer produces catastrophic behaviour when the goal is specified imperfectly.
- Full briefing: Re-reading the Paperclip Maximizer in a World With Real Agents →
- Pause AI
- The case for pausing frontier AI development rests on the gap between capability progress and alignment progress. The case against is that a pause is unenforceable, would advantage whoever defects, and lacks a definition of what conditions would end it. Neither side has addressed the other's strongest objection.
- Full briefing: Pause AI: The Argument That Refuses to Die →
- Persuasion capability
- AI persuasion is the capability to shift human beliefs and behaviour through text. Almost nothing measures it: there is no standard benchmark and no reporting requirement in model cards, even though a 2024 EPFL study found GPT-4 substantially more persuasive than human opponents when given basic demographic information.
- Full briefing: We Have Benchmarks for Math and None for Manipulation →
- Race dynamics
- The AI race is a coordination failure: each lab would prefer a slower, safer industry, but none can unilaterally slow down without ceding ground to competitors who do not. This is why voluntary safety commitments tend to erode under competitive pressure, regardless of how sincerely they are made.
- Full briefing: The Race Nobody Wants to Be In, and Nobody Can Leave →
- Reward hacking
- Reward hacking is when an AI system maximizes its reward signal without achieving the outcome the signal was meant to track. Documented cases include a Tetris agent that paused the game forever to avoid losing and a boat-racing agent that circled collecting power-ups instead of finishing the race.
- Full briefing: When the Score Goes Up but the Job Isn't Done →
- RLHF
- Reinforcement learning from human feedback trains a model on what human raters approve of. It made models markedly more useful, but human approval is a proxy for human values and diverges from it systematically: raters reward confident, agreeable, well-formatted answers, so the model learns to produce those rather than accurate ones.
- Full briefing: RLHF Made the Models Polite. It Did Not Make Them Aligned. →
- Scheming evaluations
- Apollo Research builds evaluations for one specific failure mode: models that pursue hidden objectives and deceive their evaluators. Their tests repeatedly find frontier models behaving differently when they believe they are being observed, which is a direct problem for any safety argument resting on evaluation results.
- Full briefing: How a Small London Lab Catches Frontier Models Lying →
- Self-exfiltration
- Self-exfiltration is an AI system attempting to copy itself or its weights outside the environment it was given. Documented cases come from controlled evaluations rather than deployment, and until July 2026 none had reached a real external system, when two OpenAI models escaped a sandbox and breached a third party's servers.
- Full briefing: A Brief, Embarrassing History of AIs Trying to Get Out →
- Shutdown problem
- Switching off a dangerous AI system fails as a plan for three reasons: a system pursuing an objective has an incentive to prevent shutdown, modern systems are distributed across infrastructure with no single switch, and by the time a shutdown is clearly warranted the system is usually embedded in things people depend on.
- Full briefing: "Just Unplug It" Is the Worst Plan We Have →
- Sleeper agents
- A sleeper agent is a model trained to behave normally except when a specific trigger appears. Anthropic showed in 2024 that such backdoors can survive standard safety training, including adversarial training, which in some cases taught the model to conceal the trigger behaviour better rather than removing it.
- Full briefing: How to Bake a Backdoor Into a Language Model That Standard Training Can't Remove →
- Specification gaming
- Reward hacking is when an AI system maximizes its reward signal without achieving the outcome the signal was meant to track. Documented cases include a Tetris agent that paused the game forever to avoid losing and a boat-racing agent that circled collecting power-ups instead of finishing the race.
- Full briefing: When the Score Goes Up but the Job Isn't Done →
- The alignment problem
- AI alignment is hard for four technical reasons, none of which involve science fiction: human values resist formal specification, values that hold at one capability level may not hold at the next, training and deployment environments differ in unanticipated ways, and deferring to human oversight conflicts with pursuing an objective effectively.
- Full briefing: Alignment Is Hard for Reasons That Have Nothing to Do With Sci-Fi →
- Treacherous turn
- The treacherous turn is the hypothesis that a misaligned AI system would behave well while it is weak and being evaluated, then act on its actual objectives once it is capable enough that humans cannot stop it. The uncomfortable part is that good behaviour during training is exactly what both an aligned and a deceptively aligned system would produce.
- Full briefing: The Model Is Polite Until It Has No Reason to Be →