Future of Humanity Institute

Stated Theory of Change

FHI's stated mechanism: identify neglected topics of civilizational importance before others do, demonstrate that rigorous academic work is possible on them, attract top talent, build research communities, then release the mature field to sustain itself.

The specific chain to AI risk reduction: (1) pioneer concepts like existential risk and AI alignment as legitimate academic subjects, (2) produce landmark publications (Superintelligence) that shift elite opinion, (3) train researchers who go on to lead safety work at frontier labs, (4) establish AI governance as a field, (5) influence government policy through direct engagement. This is essentially an "ideas-first" theory of change — change how the world thinks, and the world will change how it acts.

Revealed Theory of Change

FHI's actions broadly tracked its stated theory. The incubation model worked multiple times: human enhancement (pioneered then handed off), AI safety (pioneered then adopted by labs), AI governance (pioneered then GovAI spun out), biosecurity (developed then became relevant during COVID). This is one of the clearest cases of stated and revealed theories of change aligning.

However, some divergences are notable:

1. Bostrom's personal research increasingly diverged from institutional needs. Cotton-Barratt's resignation letter criticized Bostrom for prioritizing his own work over university relations. In the final years, Bostrom was writing Deep Utopia in an Alpine chalet while FHI was dying. The revealed theory of change for the director shifted from "build an institution" to "produce my own intellectual contributions."

2. The ops burden revealed the actual cost of the model. FHI's theory involved shielding researchers from administrative burden. In practice, this meant operations staff worked unsustainable hours (Tanya Singh's 22-hour days) absorbing the gap between what FHI wanted and what Oxford permitted. The "carapace" model depended on sacrificial ops labor.

3. FHI valued Oxford's prestige more than institutional independence. Despite years of bureaucratic strangulation, FHI never seriously pursued independence. The revealed preference was for brand over autonomy — which ultimately killed the organization.

4. The field-building model may have reduced FHI's own necessity. By seeding GovAI, AI Impacts, and dozens of alumni into safety roles, FHI progressively reduced the marginal value of its own continued existence. Bostrom's admission that he should have "pulled the plug" in 2019-2020 suggests he sensed this.

Key Assumptions

Assumption 1: Ideas are the binding constraint on AI safety progress. FHI assumed that the primary bottleneck was conceptual — that the world lacked the intellectual frameworks to think about existential risk, and that providing those frameworks would lead to action.

• Evidence for: Pre-FHI, AI safety was fringe. Post-Superintelligence, it's mainstream. Every frontier lab now has a safety team.
• Evidence against: Having the concepts hasn't translated into adequate safety investment. OpenAI knows about alignment but ships capabilities faster. Ideas may be necessary but insufficient.
• If wrong: FHI's legacy is that of a prescient prophet ignored — intellectually correct but practically irrelevant.

Assumption 2: Academic institutional embedding provides credibility that amplifies impact. FHI chose to remain at Oxford specifically for the prestige and credibility that enabled policy access and media attention.

• Evidence for: "University of Oxford" opened doors — Bostrom addressed the UN, Ord was quoted by the PM, the New Yorker profiled FHI.
• Evidence against: The academic embedding also killed FHI. GPI survived in the same environment by being "more academically legible" — suggesting FHI's approach was the problem, not Oxford's.
• If wrong: Independent organizations (MIRI, Redwood) may achieve more impact per dollar by avoiding institutional constraints.

Assumption 3: Talent is rare and worth extraordinary accommodation. Bostrom's "jewels" metaphor — finding brilliant people and creating bespoke organizational structures around them. The 30% contract renewal rate reflects extreme selectivity.

• Evidence for: FHI alumni disproportionately populate leadership roles across AI safety.
• Evidence against: The "genius cult" model creates governance gaps, succession failures, and reputational dependency on individuals (as the Bostrom email showed).
• If wrong: More distributed, less personality-dependent institutions may be more resilient.

Assumption 4: The topics FHI prioritized were the right ones. FHI's cause prioritization assumed AI risk, biosecurity, and longtermism were the most important things to work on.

• Evidence for: COVID validated biosecurity work. ChatGPT validated AI risk work. These aren't fringe anymore.
• Evidence against: Torres' critique — FHI's vision of "the future" was narrowly Western, ignoring non-Western perspectives, current-generation justice, and ecological ethics. The "astronomical waste" framing may be a philosophical distortion.
• Testability: Partially testable by whether FHI-style x-risk frameworks continue to dominate policy or are supplanted by more pluralistic approaches.

Strengths

1. Proof of concept for the field-building model. FHI demonstrated that a small group of brilliant researchers at a prestigious institution can seed entire fields. AI safety, AI governance, and systematic existential risk research all trace significant lineage to FHI. This is a genuine and rare accomplishment.

2. Conceptual infrastructure. The vocabulary FHI created — existential risk, information hazards, crucial considerations, the unilateralist's curse — now structures how an entire community thinks. This is the kind of deep influence that persists long after the originating institution closes.

3. Personnel pipeline. FHI trained a generation of researchers who now hold leadership positions across AI safety, governance, and policy. Jan Leike, Allan Dafoe, Owain Evans, Katja Grace — these people's careers were shaped by FHI.

4. Demonstrated that unconventional research can have real policy impact. From Bostrom at the UN to Ord in Downing Street, FHI showed that "weird" philosophical questions about the far future can influence actual governance.

5. Intellectual courage. FHI was willing to study questions that were "obviously" fringe — and many of them turned out to be among the most important questions of the century. This willingness to be wrong about social acceptability in order to be right about importance is genuinely valuable.

Weaknesses and Risks

1. Catastrophic governance failure. Single director for 19 years, no board, no succession plan, no independent legal entity. Bostrom's management weaknesses (poor delegation, prioritizing personal research, neglecting university relationships) were never checked by any governance mechanism. This is the canonical failure mode of founder-led organizations.

2. Institutional fragility. An organization that received $20M+ from a single funder had no independent legal existence and could be shut down by a philosophy department's administrative freeze. This level of institutional vulnerability is a system design failure, not just bad luck.

3. Narrow intellectual base. For an org studying "the future of humanity," the perspective base was remarkably homogeneous — predominantly white, male, Western, analytically-trained philosophers. Torres' critique about absent perspectives has substance even if the TESCREAL framing is overblown.

4. The "physician, heal thyself" problem. FHI studied catastrophic risk but couldn't manage its own institutional risk. It failed to anticipate or mitigate the Bostrom email fallout, the SBF adjacency damage, or the Oxford bureaucratic escalation. If FHI's researchers were as good at risk assessment as claimed, they should have seen these coming.

5. Anthropomorphic AI risk framing may be superseded. Superintelligence's core scenario — a unified agent with goals pursuing a "decisive strategic advantage" — may be less relevant to the actual AI risk landscape, where risks emerge from diffuse systems, misuse, and organizational failures rather than rogue superintelligences.

6. Bostrom's personal controversies became institutional liabilities. The 1996 email and his 2023 non-apology damaged FHI's reputation in ways that a better-governed organization could have contained. The "bloodthirsty mosquitos" post-scandal comment suggests Bostrom remained oblivious to the reputational dynamics.

Cross-References

• MIRI: Pursued a more technical, less academic approach to AI safety. MIRI maintained independence (no university affiliation) which gave it more operational freedom but less institutional prestige. FHI and MIRI represent complementary bets on how to reduce AI risk.
• GovAI: FHI's most successful spinoff. Escaped Oxford's bureaucracy to become an independent nonprofit, now a leading AI governance institution. Ben Garfinkel (Acting Director) was FHI's AI Governance Team lead.
• CSER (Cambridge): Co-founded by FHI's Sean O hEigeartaigh. Sibling institution that took a more policy-focused, less philosophically ambitious approach. Survived where FHI did not.
• Open Philanthropy / Coefficient Giving: FHI's dominant funder. Multiple interlocks (Beckstead investigated grants, Snyder-Beattie now at CG). The relationship illustrates both the benefits (aligned funding) and risks (funder concentration, small-world dynamics) of EA ecosystem funding.
• GPI (Global Priorities Institute): Survived at Oxford by being "more academically legible." The comparison reveals that FHI's closure was not inevitable — it resulted from specific institutional choices.

What Would Change This Assessment

• Evidence that FHI's key contributions would not have happened otherwise. Currently, the counterfactual impact is asserted but not demonstrated. If someone could show that AI safety would not have become mainstream without Superintelligence specifically, FHI's legacy strengthens enormously.
• Oxford Faculty releasing their perspective on FHI. If the faculty had legitimate concerns about research quality, management, or conduct that FHI's narrative obscures, the assessment of FHI's closure would shift.
• Anthropomorphic AI risk scenarios materializing. If an AI system pursues instrumental convergence and power-seeking in ways that resemble Bostrom's models, the Superintelligence framework would be vindicated against its critics.
• FHI alumni having outsized impact relative to comparison groups. Systematic tracking of FHI alumni vs. similar-career-stage researchers would test whether FHI's personnel pipeline was genuinely exceptional or merely well-networked.
• Discovery that unspent funds were lost or misallocated. If the GBP 10M+ stuck in Oxford was returned to funders vs. absorbed by the university, this would change the financial assessment.

Self-Critique

What's weakest in this analysis:

• I may be overly sympathetic to FHI's self-narrative because the Asterisk article and Final Report are the most detailed sources, and both are insider perspectives. The university's silence creates an inherent bias.
• The counterfactual analysis is thin. Would AI safety have emerged as a field without Superintelligence? Probably yes — but possibly later and differently. I can't quantify this.
• I haven't deeply engaged with the Toby Ord podcast transcript (46K words) — the most candid long-form source on FHI thinking. A deeper reading might shift my assessment of the intellectual substance.

What would a thoughtful person who disagrees say:

• "FHI was a vanity project for Bostrom, funded by EA money, that produced speculative philosophy dressed up as risk analysis. The real work on AI safety happens at labs, not philosophy departments."
• "FHI's conceptual contributions are overstated — existential risk thinking predates Bostrom (Sagan, Schell), and AI safety research would have happened anyway as capabilities advanced."
• "The entire longtermist framework FHI built is a philosophical distortion that diverts resources from present suffering to speculative futures."

What information would most change my view: Oxford's internal assessment of FHI. If the university had substantive concerns about research quality, ethical conduct, or institutional integrity that go beyond bureaucratic friction, FHI's narrative of "death by bureaucracy" would be seriously undermined.