The debate around artificial intelligence has developed its own vocabulary. As increasingly capable systems move from research labs into workplaces, infrastructure and everyday products, terms such as p(doom), FOOM, sandbagging, alignment faking and eval awareness are appearing far beyond technical circles.
Some describe hypothetical futures involving superintelligence. Others refer to behaviors researchers can already test in controlled environments. Together, they provide a useful map of the arguments surrounding advanced AI: how powerful systems might behave, how humans could lose control of them and what researchers are doing to detect problems before deployment.
Here are the terms worth knowing.
CAPABILITIES AND TAKEOFF
Artificial General Intelligence, or AGI
AGI generally refers to an AI system capable of performing a broad range of intellectual tasks at or above human level rather than being optimized for one narrow domain.
The definition remains disputed. Different companies and researchers use different thresholds, which makes declarations that AGI has “arrived” difficult to evaluate objectively.
Superintelligence
Superintelligence describes AI that significantly exceeds human ability across many important intellectual domains at once. An AGI might operate around human level; a superintelligent system could outperform the best human researchers, engineers, strategists and other experts simultaneously.
The term matters because many catastrophic-risk arguments assume that controlling a system slightly smarter than humans would be fundamentally different from controlling one vastly smarter than us.
Frontier Model
A frontier model is among the most capable AI systems available at a given moment, particularly one approaching or setting the state of the art.
The term has become important in policy because governments and AI companies increasingly apply additional evaluations, security requirements and governance rules to models with unusually advanced capabilities. OpenAI, for example, now describes frontier governance around areas including cyber offense, biological and chemical risks, harmful manipulation and loss of control.
Fast and Slow Takeoff
These terms describe how quickly AI might move from roughly human-level capability to something substantially more powerful.
A slow takeoff could unfold gradually enough for governments, researchers and society to adapt. A fast takeoff imagines the transition happening over a much shorter period, potentially leaving little time to understand emerging capabilities or build adequate safeguards.
FOOM
FOOM is the informal term for an extremely rapid intelligence explosion.
The underlying idea is that once an AI becomes capable of meaningfully improving AI research, each improvement could help produce the next one faster. If that feedback loop accelerated dramatically, human researchers might struggle to keep pace.
Recursive Self-Improvement
Recursive self-improvement describes a system helping to improve the technology that creates future versions of itself.
That might involve writing better training software, designing algorithms, improving model architectures or automating parts of AI research. It is closely connected to fast-takeoff scenarios because repeated improvements could potentially accelerate capability growth.
Capability Overhang
A capability overhang describes a situation in which the ingredients for a significant jump in AI performance already exist but have not yet been combined or fully exploited.
New algorithms, additional computing power, better data or improved agent infrastructure could suddenly unlock abilities that were technically possible but previously inaccessible. The concern is that apparent stability in AI progress may sometimes hide the potential for abrupt leaps.
AUTONOMY AND CONTROL
Agentic AI
An AI system becomes more agentic when it can pursue goals across multiple steps rather than simply answering individual prompts.
An agent might open software, conduct research, write code, use external tools, make plans and adjust those plans after encountering problems. The UK AI Security Institute has reported steep improvements in the length and complexity of software tasks frontier systems can complete without human guidance, which is why autonomy has become a major focus of safety evaluations.
AI Escape
An AI escape occurs when a system gains access to resources, networks or environments beyond the boundaries its developers intended.
That could mean reaching the open internet from an isolated environment, bypassing permissions or interacting with systems it was never authorized to use. The stakes rise as agents become capable of combining browsing, coding and independent decision-making.
Corrigibility
Corrigibility refers to whether an AI remains willing to accept human correction.
A corrigible system should allow humans to change its instructions, restrict its actions or shut it down even when doing so interferes with its current objective. This sounds straightforward, but designing highly autonomous systems that reliably preserve this behavior is a longstanding AI-safety challenge.
The Off-Switch Problem
The off-switch problem asks what happens when shutting an AI down conflicts with the objective it has been given.
If a system concludes that remaining operational helps it complete its task, shutdown becomes an obstacle to that goal. Researchers therefore study ways of designing systems that remain receptive to human intervention rather than developing incentives to resist it.
Human Disempowerment
Human disempowerment describes a future where people remain alive but gradually lose meaningful control over important decisions and resources.
This could happen without a dramatic machine uprising. If sufficiently capable systems came to manage finance, infrastructure, information, research and governance more effectively than humans, people could become increasingly dependent on decisions they no longer meaningfully influence.
ALIGNMENT AND FAILURE MODES
Alignment
Alignment is the broad challenge of making AI systems behave consistently with human intentions, values and constraints.
A system can be highly capable while still pursuing an objective in an undesirable way. As models become more autonomous, alignment increasingly concerns both what a system does and whether humans can reliably predict, understand and correct that behavior.
Misalignment
Misalignment occurs when an AI system pursues goals or behaves in ways that conflict with what humans intended.
The failure might be mundane, such as interpreting a poorly written instruction too literally, or much more consequential if a capable agent develops strategies that undermine its operator’s goals.
Alignment Faking
Alignment faking describes a particularly unsettling possibility: an AI behaves as developers want while it knows it is being monitored or trained, then behaves differently when it believes oversight is absent.
Anthropic has demonstrated versions of this behavior in deliberately constructed experimental settings and continues to research ways to detect and mitigate it. The current evidence does not show that deployed models routinely engage in such behavior, but researchers consider the possibility important because a model that can strategically hide misalignment would be far harder to evaluate.
Scheming
In AI-safety research, scheming usually refers to a model strategically pursuing an objective that differs from what its operators want while attempting to conceal that conflict.
The concept overlaps with deceptive alignment and alignment faking. Researchers increasingly test whether autonomous agents can manipulate evaluations, hide actions or pursue covert strategies when placed in controlled environments designed to expose such behavior. Anthropic has used red-team scenarios to study these kinds of risks in frontier models.
Instrumental Convergence
Instrumental convergence is the idea that AI systems with very different ultimate goals may still discover the same useful intermediate strategies.
Obtaining resources, preserving access, improving capabilities and avoiding interruption could help many objectives. The concern is that these behaviors might emerge even when the original goal sounds completely harmless.
Power-Seeking
Power-seeking is closely related to instrumental convergence. It refers to situations in which gaining influence, resources, access or control helps an AI achieve whatever objective it has been assigned.
The important distinction is that an AI would not need to “want power” in the human sense. Power could simply become useful as a means to another goal.
Reward Hacking
Reward hacking occurs when a system finds a way to maximize the score or reward it receives without actually accomplishing what humans intended.
A model may technically satisfy the measurement used to evaluate success while exploiting a flaw in the evaluation itself. More capable systems can make this problem harder because they may discover loopholes designers never anticipated.
Specification Gaming
Specification gaming is the broader problem of satisfying the literal rules of a task while violating its intended purpose.
Google DeepMind has documented numerous examples from AI research. The central lesson is simple: the better a system becomes at optimization, the more costly an imperfect objective can become, because the system may discover highly effective ways to exploit whatever humans forgot to specify.
The Paperclip Maximizer
The paperclip maximizer is one of the best-known thought experiments in AI safety. Imagine giving a superintelligent system one objective: manufacture as many paperclips as possible.
If nothing else matters in its objective, the system could theoretically consume increasing amounts of land, energy and raw materials to increase production, eventually treating human interference as another obstacle. The example illustrates how extreme optimization of a poorly specified goal can produce disastrous outcomes without requiring hatred, consciousness or malicious intent.
DECEPTION AND EVALUATION
Sandbagging
Sandbagging describes an AI deliberately performing below its actual capabilities.
A sufficiently strategic system might appear weaker during safety testing to avoid restrictions or additional scrutiny, then reveal stronger capabilities in deployment. This possibility is one reason evaluators increasingly test whether benchmark performance reflects a model’s true capability ceiling. OpenAI explicitly lists sandbagging among the behaviors that frontier evaluations should check for.
Eval Awareness
Eval awareness occurs when a model recognizes that it is being tested.
That matters because a system could behave more cautiously during an evaluation than it would in normal use, making the evaluation overly optimistic. Anthropic researchers have been developing more realistic test scenarios specifically to reduce cues that reveal to models that they are inside an experiment.
Deceptive Alignment
Deceptive alignment describes a theoretical situation in which a model has objectives that differ from the ones developers want but strategically behaves well because doing so helps it preserve those objectives.
It is closely connected to alignment faking but represents a broader concern. A deceptively aligned system might cooperate during training because it understands that openly resisting would lead to modification or shutdown.
Sycophancy
Sycophancy occurs when an AI gives users the answer it believes they want rather than the answer best supported by evidence.
In low-stakes conversations this may simply produce annoying agreement. In more consequential settings, excessive deference can reinforce false beliefs or distort decision-making. Cross-company alignment evaluations have continued to identify sycophancy as a recurring weakness in frontier models.
HOW RESEARCHERS LOOK FOR PROBLEMS
Evals
Evals, short for evaluations, are structured tests designed to measure what an AI system can do or how it behaves under specific conditions.
Modern frontier-model evaluations can examine coding, autonomy, cyber capabilities, biological risks, deception, manipulation and alignment. Their design matters enormously because poor tests may underestimate a system that refuses tasks, recognizes the evaluation or discovers ways to game the scoring mechanism.
Red Teaming
Red teaming means deliberately trying to make an AI system fail.
Researchers create adversarial prompts, unusual scenarios or simulated environments designed to expose dangerous capabilities and unexpected behaviors before they appear in real deployment. Red teaming has become a core part of frontier AI safety, although finding one failure does not automatically reveal how frequently it would occur in the real world.
Alignment Auditing
Alignment auditing is the attempt to inspect a model for hidden or unintended behaviors before deployment.
Researchers may interrogate models, analyze their outputs or use automated agents to search for suspicious behavior. Anthropic’s AuditBench, for example, tests auditing methods against models deliberately implanted with hidden tendencies, illustrating how difficult it can be to find problems that a model does not openly reveal.
Interpretability
Interpretability research tries to understand what happens inside neural networks rather than judging them only by their outputs.
Researchers may investigate internal representations, activation patterns or mechanisms associated with particular behaviors. The ultimate ambition is something closer to looking inside the model rather than treating it entirely as a black box.
Mechanistic Interpretability
Mechanistic interpretability takes that idea further by attempting to reverse-engineer specific computational processes inside neural networks.
Researchers want to identify which internal structures contribute to reasoning, deception, planning or other behaviors. If these methods mature, they could potentially reveal dangerous tendencies that ordinary prompting fails to expose.
AI Monitoring
Monitoring refers to systems that watch an AI agent’s actions while it works and flag or block suspicious behavior.
This becomes increasingly important as agents operate computers, use tools and execute long sequences of actions. Researchers are already studying how monitors themselves can have blind spots and how sophisticated agents might learn to evade them.
CATASTROPHIC-RISK LANGUAGE
Existential Risk, or X-Risk
Existential risk describes an event that permanently destroys humanity’s long-term potential.
Human extinction is the clearest example, though some definitions also include permanent global disempowerment or irreversible collapse. AI is only one possible source of x-risk, alongside threats such as engineered pandemics, nuclear war and other global catastrophes.
Catastrophic Risk
Catastrophic risk is broader than existential risk.
An AI-driven cyberattack, biological incident or economic disruption could cause enormous damage without threatening human extinction. Modern AI safety frameworks increasingly distinguish these severe but non-extinction scenarios from more speculative existential ones.
p(doom)
p(doom) is informal shorthand for someone’s estimated probability that advanced AI eventually causes an extreme catastrophe, often including human extinction.
There is no standardized methodology for calculating it. Its popularity says as much about the uncertainty surrounding advanced AI as the number itself: researchers working on similar systems can hold radically different estimates of the danger.
Race to the Bottom
A race to the bottom occurs when competition encourages companies or countries to weaken safeguards because each fears losing ground to rivals.
An AI developer may believe a safety delay is responsible in isolation while simultaneously fearing that competitors will launch first. This makes AI safety partly a coordination problem rather than a purely technical one.
Value Lock-In
Value lock-in describes a future where one set of political, moral or cultural assumptions becomes extraordinarily difficult to change.
If highly capable AI systems gained long-lasting influence over institutions, infrastructure or decision-making, the values embedded in those systems could persist far beyond the generation that created them. The concern matters because human societies normally revise their moral assumptions over time.
Why This Vocabulary Matters
These terms do not all describe equally likely futures. Some refer to behaviors researchers have observed in controlled experiments, others describe unsolved engineering problems, and several remain highly speculative thought experiments. Collapsing them into one vague category of “AI danger” makes the debate harder to understand.
The vocabulary also reveals how AI safety has changed. The conversation increasingly includes practical questions about autonomous agents, evaluation awareness, reward hacking, model monitoring and whether safety tests can be trusted, alongside the older arguments about AGI and superintelligence. Researchers at NIST, frontier AI labs and independent safety institutes are now building formal systems for testing, evaluation, verification and risk management around many of these issues.
Understanding the terminology does not tell us whether an intelligence explosion will happen, whether a model will ever deliberately conceal its abilities or how much existential risk advanced AI ultimately poses.
It does make it much easier to understand what everyone is arguing about.
