UK AI Security Institute (AISI)

Stated Theory of Change

AISI claims two paths to impact: (1) information provision -- build the world's best understanding of frontier AI risk and communicate it to governments so they can make informed decisions, and (2) direct mitigation -- find vulnerabilities in AI systems and work with labs to fix them, while funding independent safety research.

The implicit causal chain: AISI evaluates models -> identifies risks -> informs government -> government acts (through regulation, diplomacy, or voluntary coordination) -> AI development becomes safer. Simultaneously: AISI red-teams models -> shares vulnerabilities with labs -> labs patch them -> deployed models are marginally safer.

Irving articulated a third, deeper layer: AISI exists to "enable coordination towards safer actions" because "market forces mean that there is constant pressure to prioritize speed over caution" and "race dynamics can exacerbate risks but are not inevitable." This frames AISI as a coordination mechanism that reduces the prisoner's dilemma between competing labs.

A fourth, less visible layer emerged from Irving's interview: AISI funds theoretical research (complexity theory, information theory, game theory, formal methods) that might produce "stronger guarantees" than empirical approaches. Irving believes current safety techniques will not produce enough "nines" of reliability, and that a fundamentally different class of theoretical insight is needed. The Alignment Project's GBP 27M is a deliberate bet that these fields "have a bunch of models" that can be adapted to alignment. This is a longer-term theory of change that may take a decade or more to pay off.

Revealed Theory of Change

AISI's actions suggest a somewhat different theory than its stated mission:

What they actually optimize for:

1. Maintaining voluntary access to frontier models (this constrains everything else)
2. Producing world-class research that earns respect from the technical community
3. Building international institutional infrastructure (network of AISIs, joint evaluations, International Safety Report)
4. Growing the external alignment research ecosystem through grant-making (GBP 40M)
5. Informing UK policymakers without triggering political backlash

Where actions diverge from stated theory:

• The stated theory requires labs to cooperate, but 3 of 4 failed to provide pre-deployment access. AISI did not publicly name them or escalate the non-compliance.
• The stated theory assumes evaluation results drive government action, but no legislation has followed two years of increasingly alarming findings (cyber capabilities doubling every 8 months, universal jailbreaks on every model).
• The rebrand from "safety" to "security" narrowed scope precisely as AISI's own research showed risks were broadening.
• The simultaneous Anthropic MOU announcement with the rebrand suggests AISI is being used as a bridge for tech company partnerships with the UK government, not just as a safety body.
• Irving is surprisingly candid that the labs get value from AISI's work: "they want usually to have us give them correct information." This means AISI functions partly as a free, government-funded red team for private companies.

The revealed theory of change is closer to: AISI builds technical credibility and international legitimacy now, so that when a crisis or political window opens, the institutional infrastructure exists to act. This is a bet on future leverage rather than current enforcement.

Key Assumptions

1. Voluntary cooperation provides sufficient model access.

• Evidence against: 3 of 4 labs failed to honor pre-deployment commitments. Access is typically API-level, not weights. Labs can observe evaluations on their servers. Irving acknowledged the UK "dropped its request for model weights after labs pushed back."
• Evidence for: Irving says the voluntary regime is "working decently well in the sense that developers have made commitments and they're continuing to follow many of those." Red team has tested 30+ models. Irving noted open-weight model research can supplement understanding of where white-box access matters.
• Testable: Track compliance rates over time.
• If wrong: AISI's evaluation work becomes progressively less meaningful as it evaluates only what labs choose to share, when they choose to share it. Eval awareness (which Irving confirms is "increasing fairly rapidly" and getting worse with newer models) makes API-only evaluation increasingly unreliable.

2. Benchmarks can meaningfully predict real-world model behavior.

• Evidence against: "Beyond Benchmarks" paper argues this is infeasible for deep learning. Ada Lovelace identifies the bait-and-switch problem. Irving himself: "The science of evaluations is not strong enough that we can confidently rule out all risks."
• Evidence for: AISI's evaluations have found genuine vulnerabilities that labs have fixed -- including pre-deployment fixes to live models, not just future versions. Capability trend data (cyber doubling times) provides useful macro-level information even if individual evals don't certify safety. Irving's team also does human-in-the-loop evaluations (including literal wet lab experiments for bio) that go beyond benchmarks.
• Testable: Track whether AISI-evaluated models cause fewer real-world incidents than un-evaluated ones.
• If wrong: AISI becomes an expensive provider of false assurance. This is the most dangerous failure mode.

3. Findings will translate into action.

• Evidence against: Two years of increasingly alarming data have not produced legislation, enforcement mechanisms, or even public naming of non-compliant labs.
• Evidence for: AISI has influenced at least some lab behavior (jailbreak patching, and Irving claims iterative fixes to deployed models not just future ones). The Alignment Project funds real research. International coordination has expanded. Irving noted that government briefings are "changing on the margin" but officials "have other priorities."
• Testable: Track policy actions following AISI reports.
• If wrong: AISI becomes an extremely well-informed observer of a catastrophe it lacks the power to prevent.

4. Political will persists.

• Evidence against: The safety-to-security rebrand, alignment with Trump-era US anti-safety rhetoric, refusal to sign Paris Statement, UK and US attempting to rename the International Network of AISIs (rejected by other members). The Labour government's messaging prioritizes "unleashing AI" over managing risks.
• Evidence for: GBP 240M committed for 2026-2030. Alignment Project expanding. Irving and Leung remain in post. Irving says "the UK government does care a lot about these risks."
• Testable: Budget allocations, leadership stability, scope of research agenda.
• If wrong: AISI is hollowed out or redirected toward serving industry interests rather than public safety.

Strengths

Talent density. AISI has assembled what is likely the most technically sophisticated AI safety team in any government. Irving, Leung, Summerfield, and Gal have published at the absolute frontier. The red team's jailbreaking record (100% success rate against all models) demonstrates genuine capability. Irving's 2-hour Cognitive Revolution interview is the most technically substantive public statement on AI risk from any government official anywhere.

Pure government funding. No lab money, no philanthropic strings, no equity holders. This is structurally unique among major AI safety organizations and eliminates the financial conflict of interest that compromises most others.

Open-source infrastructure. Inspect has become the standard evaluation framework. By creating tools others adopt, AISI multiplies its impact far beyond its own testing. This is AISI's highest-leverage output -- shaping how the entire world measures AI risk.

Grant-making at scale. GBP 40M in alignment/safety grants, drawing 800+ applications from 42 countries, makes AISI the largest government funder of alignment research. Irving is deliberately targeting theorists from complexity theory, game theory, and cognitive science who would not otherwise work on alignment. This diversifies the alignment research ecosystem.

Frontier intelligence. The Frontier AI Trends Report provides genuinely alarming, data-driven public intelligence that no private entity would publish. This is a pure public good.

First-mover institutional advantage. AISI created the template for national AI safety institutes now adopted by 7+ countries. The international network and joint evaluation protocols create lock-in effects that survive political changes.

Research quality. Publication in Science (persuasion study), 10 NeurIPS papers, and active engagement on LessWrong/Alignment Forum demonstrate that AISI researchers are producing genuinely novel work, not just running standardized compliance tests.

Weaknesses and Risks

No enforcement power. This is the fundamental weakness. AISI can identify problems but cannot compel action. A pharmaceutical safety body without the power to block drugs would not be considered meaningful regulation. The voluntary framework has already demonstrably failed (3 of 4 labs non-compliant). Ada Lovelace: "A testing regime is only meaningful with pre-market approval powers underpinned by statute."

Political vulnerability. The rebrand demonstrates that a single political decision can narrow AISI's scope. Being inside DSIT rather than an independent statutory body means AISI can be redirected overnight without legislation. The US AISI faced exactly this threat under Trump -- the UK AISI could face the same.

Evaluation theatre risk. If benchmarks fundamentally cannot predict real-world model behavior (the "Beyond Benchmarks" argument), AISI's evaluations may provide false assurance rather than genuine safety. Labs may use "AISI-tested" as a stamp of legitimacy for deployment decisions. Alfour's argument is specific: "Evals move the burden of proof away from AI Corporations" -- making labs appear responsible without constraining them.

Revolving door. Every senior technical leader came from a lab they now evaluate. This creates both the competence that makes AISI functional and the cultural capture that may limit its adversarial stance. The TIME profile was explicit: keeping labs "friendly and collaborative" takes priority over confrontation because AISI depends on voluntary access.

Scope narrowing under political pressure. The vocabulary shift (dropping "bias," "discrimination," "accountability") and security focus may leave AISI unable to address the societal-scale harms that actually affect most people. The BRAID critique -- that safety is a social problem, not a technical one -- remains valid. South Korea's AISI Director pushed back: "We are doing science -- and safety risks don't suddenly disappear."

Informing without acting. Two years of data showing universal jailbreaks, rapid capability growth, and lab non-compliance have not produced visible policy change. There is a risk that AISI becomes a very expensive early warning system connected to nothing.

Eval awareness is accelerating. Irving confirmed: "The newer models are more eval aware than the previous models. So that's increasing fairly rapidly. And the degree to which you can mitigate that is unclear at this stage." This means AISI's core evaluation methodology faces a moving target that may outpace its countermeasures.

Cross-References

• METR: Independent eval org doing similar frontier model testing outside government. Complementary rather than competing -- METR can be more adversarial because it doesn't depend on lab goodwill for model access in the same way. No direct comparison between their methodologies exists publicly.
• Anthropic / OpenAI / DeepMind: The labs AISI evaluates. The power asymmetry is extreme -- labs control access, AISI provides free bug-finding services. The Alignment Project creates a healthier dynamic where AISI funds research that benefits the field. Irving's personal connection to these labs is deep (former staff at both OpenAI and DeepMind; he called Paul Christiano his "favorite collaborator").
• GovAI (Centre for the Governance of AI): Intellectual progenitor. Leung co-founded GovAI, Dafoe moved to DeepMind. The GovAI -> AISI pipeline shows how academic governance research becomes government policy.
• Ada Lovelace Institute: AISI's most constructive institutional critic. Their proposed statutory framework (mandatory pre-deployment testing, FOIA transparency, enforcement powers) is essentially the blueprint for what AISI would need to become effective.
• ControlAI: Represents the "evals are harmful" position -- that AISI's approach crowds out the binding regulation that would actually constrain labs.
• ARC (Alignment Research Center): Christiano (founder) is on AISI advisory board. ARC's theoretical research agenda (heuristic estimation, opposite arguments in debate) directly connects to AISI's alignment funding priorities. Irving cited ARC's work on formalizing heuristic reasoning as a key area.
• US CAISI/NIST: AISI's main international partner. Both renamed from "safety" to "security" in 2025. The US body has been more severely defunded and deprioritized under the Trump administration. Christiano moved from ARC advisory board to US AISI.
• Redwood Research: Shlegeris on Alignment Project advisory board. AI control research (which AISI's ControlArena supports) is a key area of overlap.

What Would Change This Assessment

Upward:

• UK enacts an AI Bill granting AISI statutory authority to compel model access and block unsafe deployments. This would transform AISI from advisory to regulatory.
• AISI publicly names a lab that failed to provide model access and triggers consequences. This would demonstrate independence from industry.
• An AISI evaluation discovers a previously unknown dangerous capability that leads to a deployment decision being reversed. This would validate the theory of change.
• The Alignment Project produces a theoretical breakthrough (especially in the debate, complexity theory, or learning theory domains Irving prioritizes) that fundamentally advances safety research.
• Evidence emerges that AISI evaluations have privately blocked or delayed specific deployments. If this has happened behind closed doors, AISI's practical impact is significantly higher than the public record suggests.

Downward:

• AISI evaluates a model, finds it acceptable, and that model subsequently causes significant real-world harm. This would demonstrate evaluation theatre.
• Further budget cuts, scope narrowing, or leadership departures (Irving or Leung) that hollow out technical capability.
• Labs further withdraw from voluntary cooperation, reducing AISI to evaluating only what labs want evaluated.
• The international AISI network fragments as the US/UK "security-only" position drives away other countries.
• Open-source models reach frontier capability levels without any governance framework, rendering AISI's lab-partnership model obsolete.

Self-Critique

Weakest claim: My assessment that the rebrand represents a genuine narrowing rather than cosmetic rebranding. AISI insiders maintain the work hasn't changed, and Irving's March 2026 podcast covers the full spectrum of risks including loss of control with no evident narrowing. The vocabulary shift in public materials may not reflect internal research priorities. However, the Opinio Juris documentation of systematic vocabulary changes is hard to dismiss as purely cosmetic.

Potential bias: I may be too sympathetic to the "Beyond Benchmarks" critique. Evaluations may provide more useful information than the paper's theoretical argument suggests, even if they can't provide formal guarantees. The practical value of finding and patching jailbreaks is real even if it doesn't constitute "regulation" in the traditional sense. Irving's point about human-in-the-loop evaluations (wet lab experiments, conversational domain assessments) shows AISI goes beyond simple benchmarking in ways the critique doesn't fully address.

What I should have checked but could not: Hansard parliamentary debates about AISI (blocked by 403 errors). OpenAI's perspective on AISI collaboration (website blocked). Internal staff views post-rebrand. Detailed budget breakdown. Specific deployment decisions influenced by AISI evaluations. Comparison with METR's methodology and results.

What a thoughtful disagreer would say: "AISI is doing the best it can within the constraints of democratic government. Building technical credibility now is the right strategy because enforcement power requires political consensus that doesn't yet exist. The rebrand is pragmatic adaptation to political reality, not surrender. And the alternative -- no government AI safety body at all -- is clearly worse. You are holding AISI to the standard of a regulator when it was never designed to be one."

This is a fair point. The question is whether "doing the best it can" within a fundamentally constrained structure is good enough given the pace of AI development. Irving himself seems uncertain about this -- his optimism about solvability ("in 50 years, 100 years, 1000 years someone will have solved alignment") is paired with deep uncertainty about timing ("hopefully it will have been us kind of in time").

Single weakest claim: That AISI's grant-making (GBP 40M) will produce meaningfully better alignment research than would have happened anyway. The counterfactual is hard to assess. However, Irving's explicit goal of attracting theorists from complexity theory and game theory -- people who would not normally work on alignment -- suggests some genuine additionality. Whether this pans out is a bet.

Information that would most change my view: Evidence that AISI evaluations have actually blocked or delayed a specific deployment. If this has happened behind closed doors, the entire assessment of AISI's practical impact would need to be revised upward significantly.