Future of Humanity Institute

Status: CLOSED (April 16, 2024, after 19 years)

Theory of Change

FHI's theory of change was field-building through intellectual incubation. In their own words:

"We attempted to notice overlooked topics deserving of research and attention early, germinating it in the sheltered FHI greenhouse; showing that progress could be made; coalescing a field and setting research directions; attracting bright minds to it; and once it's established enough, setting it free, and moving onto the next seedlings." — Anders Sandberg, Final Report

The causal chain: (1) identify neglected topics of existential importance, (2) demonstrate rigorous research is possible on them, (3) attract talent and funding, (4) build a research community, (5) spin the field off when it can sustain itself, (6) move to the next neglected topic.

This was enacted across multiple domains: human enhancement ethics (2005-2010), existential risk as a systematic category (2002-2020), AI safety and alignment (2011-present), AI governance (2017-2021), biosecurity (2016-2020), digital minds (2020-2024).

Bostrom articulated the research philosophy early: "We pursue an opportunistic research agenda, in that we focus on those questions that are amenable to analysis, and where new insights would bring the greatest payoff in terms of improving humanity's ability to navigate its future" (April 2006).

What They Did

Intellectual output. FHI coined or formalized an extraordinary number of concepts now standard in AI safety and EA: existential risk (as systematic category), information hazards, the unilateralist's curse, existential hope, crucial considerations, the vulnerable world hypothesis, anthropic shadow, the simulation argument, the paperclip maximizer, comprehensive AI services. The terminology itself shaped how the field thinks.

Books. Superintelligence (Bostrom, 2014) became a NYT bestseller, was praised by Musk and Gates, and catalyzed AI safety as a serious field. The Precipice (Ord, 2020) quantified existential risks and directly influenced UK and UN policy. Deep Utopia (Bostrom, 2024) explored the positive scenario.

Organizations seeded. GovAI (Centre for the Governance of AI), AI Impacts, Giving What We Can, the EA movement's intellectual foundations, LessWrong (via Overcoming Bias), CSER (co-founded by FHI's Sean O hEigeartaigh). The Rationalist community traces significant lineage to FHI-associated work.

People trained. Jan Leike (created RLHF, led alignment at OpenAI, now at Anthropic). Allan Dafoe (founded AI governance as a field, now Google DeepMind). Owain Evans (TruthfulQA benchmark). Katja Grace (AI Impacts, expert surveys). Stuart Armstrong (Aligned AI). Andrew Snyder-Beattie (now Coefficient Giving biosecurity). Leopold Aschenbrenner (interned at FHI, later at OpenAI).

Policy influence. Bostrom briefed the UN General Assembly (2015). Multiple staff presented to UK Parliament. Ord contributed to the UN 2020 Human Development Report. Ord was quoted by the UK PM at the UN General Assembly (2021). FHI co-wrote the "Future Proof" UK resilience report.

COVID response. The biosecurity team launched an Epidemic Forecasting Project reaching 10,000 users/day. Brauner et al. (2021) on non-pharmaceutical interventions became one of the ten most-cited papers in Science's history.

Prolificacy. In its first 18 months alone (2005-2007): 40 journal papers, 43 articles and chapters, 5 books — with only 3 salaried researchers plus the director. Peak team size was approximately 40 (2019-2020).

Key People

Nick Bostrom — Founding director (2005-2024). Swedish philosopher, PhD LSE. Described by a colleague as having "the long tail of intelligence" that was visible only in discussion. Worked nights, maintained a mysterious FHI "scorecard" on the whiteboard. Brilliant researcher but poor delegator who "prioritized his own research over FHI's relationship with the university" (per Cotton-Barratt's resignation letter). Post-FHI: founded Macrostrategy Research Initiative. No longer Oxford faculty.

Anders Sandberg — First hire (2006), Senior Research Fellow. Bostrom's intellectual complement — "ebullient and whimsical" to Bostrom's "ultra-serious." Authored the Final Report, the institutional memory of FHI. Now at Mimir Center in Stockholm.

Toby Ord — Senior Research Fellow. Computer science turned philosopher. Founded Giving What We Can, co-founded EA. Published The Precipice, the most policy-impactful FHI output. Was notably skeptical of SBF/FTX involvement. Post-FHI: AI governance at Oxford Martin School.

Notable departures: Owen Cotton-Barratt resigned August 2021, criticizing Bostrom's management in a letter that leaked to the faculty. Tanya Singh (senior ops, sometimes worked 22-hour days) departed June 2022 after the hiring freeze. GovAI's entire team spun out in 2021 "to escape bureaucratic restrictions."

Money and Incentives

Total identifiable funding (2005-2024): Approximately $30-35M over 19 years from all sources.

Revenue breakdown by source:

Open Philanthropy: ~$20.9M (2016-2021) — dominant funder post-2016. Flagship grant: $12.25M (2018) for global catastrophic risks. Several grants investigated by Nick Beckstead, himself a former FHI fellow.
James Martin / Oxford Martin School: ~GBP 2.5M (2005-2016, 60% of early funding)
Amlin (reinsurance): ~GBP 0.9M (2013-2016)
European Research Council: EUR 2M Advanced Grant (2015)
FLI/Musk: ~GBP 1M+ (2015)
Survival and Flourishing Fund: ~$250K
Private donors: ~GBP 760K pre-2016

Business model: 100% grant/donation-dependent. No commercial revenue, no government contracts as a regular stream. All funds flowed through Oxford University. Staff on Oxford pay scales.

The Oxford money trap: By 2020, FHI had approximately GBP 10M in the bank — much of it from the unspent 2018 Open Philanthropy grant — but could not spend it because the Faculty of Philosophy had frozen hiring and fundraising. The amount ultimately returned to Open Philanthropy or disposed of after closure has not been publicly disclosed.

Institutional structure: FHI was a research centre within the Faculty of Philosophy, University of Oxford. No independent legal entity, no independent board, no independent hiring authority. This made FHI entirely dependent on the faculty's goodwill for operational decisions. GPI (Global Priorities Institute, a similar Oxford group that survived) was "less zany and more academically legible."

Funder concentration risk: Open Philanthropy provided roughly 60-70% of total post-2016 funding. Nick Beckstead (grant investigator for several major OP grants) was himself a former FHI research fellow, illustrating the tight interlocks between FHI and the EA funding ecosystem.

Key incentive dynamic: Bostrom and senior researchers wanted to "move fast, hire quickly, work with industry." Oxford wanted standard academic procedures, teaching contributions, and legible philosophical output. As Bowerman put it: "The philosophy faculty's currency is peer-reviewed papers in prestigious journals. That wasn't the currency of FHI. The currency was cool ideas that could improve the world." This mismatch, never resolved, was fatal.

What Others Say

The "physician, heal thyself" critique. Torres: "Longtermists failed to anticipate and mitigate the risks posed by Sam Bankman-Fried... failed to anticipate and mitigate the negative press that buying a huge castle would cause... weren't even able to keep FHI from being shut down. How much evidence do we need that longtermists aren't good at anticipating and mitigating risks?"

The narrowness critique. Torres: "The utopian vision that FHI developed is thoroughly Western, Baconian, and capitalistic... designed almost entirely by extremely privileged white men at elite universities and in Silicon Valley... conspicuous absence of virtually any reference whatsoever to what the future could or should look like from the perspectives of Indigenous communities, Afrofuturism, feminism" and other non-Western traditions.

The anthropomorphism critique. Quillette's review of Superintelligence after 10 years: "Many of Bostrom's arguments rest upon profoundly anthropomorphic premises... Many who work with code find the prospect of programs becoming goal-seeking, power-seeking, and 'making their own decisions' fundamentally implausible." Drew McDermott's classic critique is cited: calling a program function "GOAL" does not give it goals.

The EA/inequality critique. Jacobin: EA and FHI are "at heart a conservative movement, which attempts to present billionaires as a solution to global poverty rather than its cause." Nick Beckstead's claim that "saving a life in a rich country is substantially more important than saving a life in a poor country" is cited as representative.

The methodological critique. Reflective Altruism's epistemics series demonstrates that FHI/EA policy proposals (like strengthening the Biological Weapons Convention) are "substantially more complex than they appear" and that "diamonds in the rough are rare."

The balanced view. Andrew Maynard (Future of Being Human): FHI was "effective in pushing the bounds of thinking in ways that transcended disciplines" but its thinking placed "far greater emphasis on philosophical speculation than practical reality." The world needs boundary-spanning thinking, but grounded in humility and inclusive of diverse perspectives.

Defenders. Moorhouse: "Especially for its size, FHI was staggeringly influential." Even critics rarely dispute FHI's conceptual influence — the debates are about whether that influence was net positive.

What's Absent

No published annual financial reports in 19 years of operation
No diversity reporting or systematic engagement with non-Western perspectives on the future
Oxford Faculty of Philosophy never gave its side of the story beyond a boilerplate statement
No formal impact evaluation of FHI's counterfactual contribution
No succession plan despite 19 years of single-director governance
No published data on how the GBP 10M+ unspent grant money was ultimately disposed of
No systematic alumni tracking — claims about influence rely on anecdote
No explanation for why GPI survived identical Oxford constraints while FHI did not