Centre for the Governance of AI (GovAI)

Stated Theory of Change

GovAI's stated theory is unusually explicit and thoughtful. Dafoe's 2020 essay argues that ~80% of AI governance research value comes from field building -- "improving insight, expertise, connections, and authority" -- rather than from specific research products. The mechanism: train and credential experts who care about AI's long-term impacts, then place them at consequential institutions (governments, labs, think tanks) so they are "in the room" when critical decisions are made.

This is the "Metropolis" model -- build dense connections to policy, social science, and computer science rather than an "isolated Island" focused narrowly on alignment. GovAI explicitly covers structural risks (inequality, totalitarianism, race dynamics) alongside the safety risks that dominate most AI safety organizations.

The causal chain: Research + fellowships -> trained governance experts -> placed at influential institutions -> better decisions about AI deployment/regulation -> reduced probability of catastrophic outcomes.

Revealed Theory of Change

GovAI's actions are remarkably consistent with its stated theory. The fellowship programs process ~90+ people per year through structured governance training. The alumni pipeline is the revealed priority: Jade Leung (UK AISI CTO), Allan Dafoe (DeepMind), Lennart Heim (RAND), Helen Toner (CSET), Markus Anderljung (EU Code of Practice drafter). Every major GovAI leadership figure has either moved to or maintained connections with consequential institutions.

The research agenda reveals a secondary priority: producing enough credible work to maintain the organization's intellectual authority and to justify the talent pipeline. Publications in Science, Nature Machine Intelligence, and Lawfare serve as credentialing vehicles as much as policy instruments.

One potential divergence: GovAI's stated emphasis on structural risks (inequality, democracy, totalitarianism) is broader than what its revealed research priorities suggest. The actual research output skews heavily toward frontier model regulation, compute governance, and risk management -- topics with more direct connection to the longtermist/x-risk community that funds them than to the structural risks Dafoe articulates.

Key Assumptions

1. "In the room" assumption: Having governance-minded people at labs and governments will actually improve decisions.

• Evidence for: Dafoe reports having "a lot of potential impact" advising DeepMind leadership. Jade Leung's position at UK AISI gives governance thinking institutional weight. Anderljung's EU Code of Practice role is direct policy influence.
• Evidence against: Labs' structural incentives (commercial competition, investor returns) may override any individual adviser's influence. People placed at labs may gradually adopt the lab's perspective. No public evidence that any GovAI alumnus has actually changed a consequential decision at a lab.
• Testable: Yes, but difficult. Would require tracking specific decisions where governance considerations were raised by GovAI alumni.
• If wrong: GovAI becomes an expensive feeder program for AI lab hiring without governance impact.

2. Talent scarcity assumption: The AI governance field is so small that adding trained people is highly valuable.

• Evidence for: Garfinkel's claim that fewer than 10 people have combined regulatory and AGI expertise seems credible. The frontier of knowledge on most governance topics is "one person and two months of work."
• Evidence against: As AI governance becomes more prominent, the talent pool expands from conventional routes (law schools, policy programs, think tanks) without GovAI's intervention. The marginal value of additional GovAI fellows may be declining.
• If wrong: GovAI's fellowship programs become redundant as conventional institutions produce governance talent.

3. Safety tractability assumption: Garfinkel argues catastrophic AI safety risks are the most tractable long-term concern because "everyone basically is on the same page globally that we would like for AI systems to do the things that we want them to do."

• Evidence for: RLHF, constitutional AI, and other alignment techniques were partly developed by safety-motivated researchers.
• Evidence against: Garfinkel himself notes the equilibrium argument that the "market would otherwise have provided" safety improvements. The tractability of structural risks (inequality, authoritarianism) through governance research is less clear.
• If wrong: GovAI's focus on safety-adjacent governance may be less impactful than structural interventions.

4. Funder independence assumption: Near-total dependence on CG/Open Phil funding does not compromise research independence.

• Evidence for: Garfinkel's publicly skeptical views on AI risk -- lower than the CG/Open Phil median -- suggest intellectual independence. The CIP Windfall Clause critique co-authored by a GovAI researcher suggests willingness to criticize internal proposals.
• Evidence against: Ajeya Cotra on the advisory board while at the primary funder is a structural conflict. No examples of GovAI research that contradicts CG/Open Phil priorities. Research topics cluster within the longtermist frame that motivates CG funding.
• If wrong: GovAI's research agenda is shaped by what its funder wants to see rather than what the field needs.

Strengths

Intellectual honesty. Garfinkel's public skepticism of classic AI risk arguments, while directing an AI governance organization funded by people who believe in those risks, is genuinely unusual. His demand for rigor -- "if the entire case for treating AIs and existential risk hung on these blog posts, I wouldn't really feel comfortable advocating for millions of dollars" -- sets an epistemic standard that few organizations in this space match.

Talent pipeline results. The alumni list speaks for itself. Jade Leung at UK AISI, Dafoe at DeepMind, Heim at RAND, Toner at CSET -- these are genuine placements at institutions that make consequential decisions about AI.

Intellectual breadth. The structural risks framework (misuse + accidents + structural) is more comprehensive than the narrow alignment focus of many AI safety organizations. The cooperative AI agenda addresses a real gap.

Compute governance research. This is GovAI's most distinctive and policy-relevant contribution. The compute governance framework has influenced real policy (US chip export controls, EU AI Act thresholds).

Policy engagement. Operating across US (NIST consortium, DC fellowship, CNAS), UK (Cabinet Office secondment, AISI network), and EU (GPAI Code of Practice) simultaneously is rare and valuable.

Weaknesses and Risks

Near-total funder dependency. $19.5M from CG/Open Phil with no significant alternative funding sources. If CG shifts priorities -- as happened when it scaled back Open Phil's longtermist giving in 2023 -- GovAI could face existential financial pressure. The no-for-profit-donations policy limits diversification.

Revolving door problem. GovAI's theory of change (place people at labs) is structurally in tension with its role as an independent governance voice. Dafoe's dual role as GovAI President and DeepMind Director is the clearest example. Can GovAI credibly critique DeepMind's governance practices while its founder-President works there?

Financial opacity. No financial filings of any kind for an organization receiving nearly $20M. Budget, salaries, spending categories -- all unknown. For an organization that advocates for governance transparency, this is a credibility gap.

Echo chamber risk. All funding comes from EA/longtermist sources. The advisory board is entirely from this ecosystem. Staff backgrounds cluster in philosophy, economics, and political science from elite Anglophone universities. Engagement with the AI ethics / FAccT community is minimal.

Impact measurement vacuum. GovAI has never published an evaluation of whether its theory of change is working. The claims that alumni placement = governance influence are plausible but unsubstantiated.

Scale mismatch. 15-25 core staff trying to influence multi-trillion-dollar industries and dozens of national governments. GovAI's influence is necessarily mediated through personal networks, which means it could be disproportionately affected by key person departures.

Cross-References

vs. CSET (Georgetown): Both are AI governance research organizations, but CSET focuses more on US national security applications and data-driven analysis. Helen Toner bridges the two. CSET has deeper government connections; GovAI has broader international scope.

vs. CLTR (Centre for Long-Term Resilience): CLTR focuses specifically on UK policy and institutional resilience. GovAI is more international and more research-oriented. Alumni overlap (Jess Whittlestone, Hamish Hobbs at CLTR are GovAI affiliates).

vs. MIRI: Opposite intellectual approaches. MIRI pursues narrow alignment theory with shutdown advocacy; GovAI builds a broad governance community with structural risk emphasis. Garfinkel has explicitly criticized the intellectual foundations MIRI relies on.

vs. Coefficient Giving: GovAI is functionally a CG portfolio organization. Ajeya Cotra on both sides of the relationship. The Coefficient Giving forum post (abergal) explicitly lists GovAI as an example organization whose "marginal hires could found capacity-building orgs." CG views GovAI partly as a source of trained people.

vs. Cooperative AI Foundation: Spun out from GovAI research, separate entity with $15M endowment. Lewis Hammond (CAIF Acting ED) is a GovAI affiliate. The relationship demonstrates GovAI's institutional generativity.

What Would Change This Assessment

• Publication of financial data (990 filings, UK accounts) showing budget allocation and salary levels would significantly improve assessment of whether funding is being used effectively.
• Evidence of alumni actually changing a decision at a lab -- a specific case where a GovAI alumnus pushed back against a deployment decision, changed a policy position, or shaped a regulatory framework in a traceable way.
• A substantial published critique of GovAI specifically (not just the broader ecosystem) by a credible external analyst would help calibrate the assessment.
• Funding diversification -- if GovAI secured significant funding from government grants, non-EA foundations, or other non-longtermist sources, it would reduce the dependency concern.
• Evidence that alumni at labs are NOT maintaining governance orientation -- reports of GovAI alumni becoming capabilities-focused or pro-deployment without safety conditions would undermine the core theory of change.
• Failure of compute governance approach -- if algorithmic efficiency improvements make compute governance obsolete faster than expected (the Rob Wiblin pushback in the Heim podcast), GovAI's most distinctive contribution loses relevance.

Self-Critique

Weakest claim: The assessment that GovAI's funder dependency "does not compromise research independence" based on Garfinkel's skeptical views. This is indirect evidence. It is entirely possible that GovAI's research agenda is shaped by CG priorities in ways that are invisible from the outside -- not through explicit pressure but through selection of which topics to pursue and which to avoid.

Potential bias: This analysis may underweight GovAI's actual policy impact because impact through personal influence is inherently difficult to observe from outside. The brief relies on publicly visible outputs (papers, placements, policy roles) and may miss private advising, behind-the-scenes influence, and informal network effects.

What a thoughtful critic would say: "GovAI is the EA/longtermist community paying itself to train people to spread EA/longtermist ideas to government and industry. The 'field building' theory of change is unfalsifiable -- any outcome can be claimed as building the field. The alumni 'placed at labs' are being absorbed by industry, not influencing it. And the financial opacity suggests an organization that is accountable to its funder, not to the public whose governance it claims to serve."

Information that would most change my view: Access to internal GovAI communications or alumni surveys showing whether/how former fellows maintain governance orientation at their subsequent employers. Also, the 2024 annual report content would significantly update understanding of current priorities and scale.