Machine Intelligence Research Institute (MIRI)

Stated Theory of Change

MIRI's stated theory of change has two layers:

The macro thesis: Building artificial superintelligence with current understanding will cause human extinction (>90% probability). The alignment problem is unsolvably hard on the timeline available. Therefore, the only viable path is an international moratorium on frontier AI development, lasting decades or generations, until humanity develops sufficient understanding to proceed safely.

The micro theory (how MIRI contributes): MIRI produces candid, uncompromising communications that reach policymakers and the public. This shifts the Overton window. Simultaneously, MIRI's technical governance team produces concrete policy proposals (treaty drafts, compute monitoring schemes, governance research agendas) that give policymakers actionable options. Together, these create the conditions for an eventual moratorium.

The causal chain: MIRI communications -> public awareness -> political will -> international coordination -> moratorium -> time for alignment research -> safe AI development eventually.

Revealed Theory of Change

Looking at where MIRI actually spends money and attention, several divergences emerge:

The real theory is "keep the alarm sounding until reality catches up." MIRI doesn't have a credible political pathway to a moratorium. Nate Soares acknowledged "my sense of the appetite right now isn't very high." The governance team's work is serious but there's no lobby, no political coalition, no state-level allies championing their proposals. The actual function is: keep the most extreme version of the risk argument in public discourse, so that when (if) AI systems do something alarming, there's an existing framework people can turn to.

The Yudkowsky personality engine. Whatever the org chart says, MIRI's media impact depends almost entirely on Yudkowsky's personal brand and Soares' complementary measured persona. The book, the documentary, the podcast appearances -- these are personality-driven, not institution-driven. The 2024 comms strategy explicitly acknowledges being "bottlenecked on his time, energy, and endurance."

Research as vestige. The $2.3M research budget is the smallest category, and the three-way split between Eliezer/Nate/Malo suggests no coherent research direction. The research function appears to serve institutional identity rather than any expectation of breakthroughs.

The governance team is the most promising element. The TGT's work on compute governance, treaty frameworks, and scenario analysis is the most concretely useful thing MIRI produces. It was founded by Lisa Thiergart (who then left), suggesting both its promise and the difficulty of retaining talent.

Key Assumptions

1. Alignment is unsolvably hard on current timelines.

• Evidence for: MIRI's own research failed after 7+ years. Tsvi says "0% solved." The field has no consensus on a viable alignment approach for ASI.
• Evidence against: Modern AI systems exhibit much more tractable alignment properties than MIRI predicted (see the Asterisk review). Interpretability is advancing. RLHF works better than expected. Paul Christiano and others think incremental progress is meaningful.
• Testable? Partially -- continued AI progress without catastrophe would weaken this claim.
• If wrong: MIRI's entire pivot was unnecessary and they abandoned their most valuable work.

2. Fast takeoff / intelligence explosion will occur.

• Evidence for: AlphaGo Zero's rapid progress. Theoretical arguments about recursive self-improvement.
• Evidence against: AI progress has been continuous and predictable, following scaling laws. AI 2027 and most alignment researchers now expect gradual, continuous improvement. Yudkowsky predicted AI would spend "almost no time" between village idiot and Einstein; current AI has been in that zone for years.
• Testable? Yes -- observe whether AI progress is continuous or discontinuous.
• If wrong: Most of MIRI's urgency arguments collapse. Gradual progress allows iterative alignment work.

3. International coordination for a moratorium is the most viable path.

• Evidence for: Nuclear non-proliferation partially worked. Climate agreements exist. China has signaled some openness to AI governance.
• Evidence against: No geopolitical precedent for halting a lucrative technology during peacetime. US administration actively hostile to AI regulation. DeepSeek showed China is competing hard. The 2025 fundraiser falling short of target suggests even MIRI's donors are skeptical.
• Testable? Watch for any government to seriously propose a moratorium.
• If wrong: MIRI's policy advocacy is tilting at windmills.

4. MIRI's "moral clarity" approach is more effective than incremental policy engagement.

• Evidence for: The book became a NYT bestseller. MIRI claims their message reaches audiences other orgs can't.
• Evidence against: No policy outcomes. The "Doomer's Dilemma" article documents the political backlash against doomers. EU rebranded "AI Safety Summit" as "AI Action Summit." US NIST told scientists to eliminate "AI safety" language. The Overton window may be closing, not opening.
• Testable? Track policy outcomes over 2-3 years.
• If wrong: MIRI is spending its remaining runway on communications that don't produce policy change.

Strengths

Genuine independence. Zero financial ties to AI labs. This is almost unique among AI safety orgs and means their positions are not compromised by funder incentives.

Intellectual honesty about failure. The 2020 research failure admission and the transparent strategic pivot are rare in any organization. Most would quietly rebrand rather than publicly say "our core research program failed."

Deep bench of accumulated thinking. 25 years of reasoning about alignment gives MIRI a perspective that newer orgs lack. Concepts they pioneered (corrigibility, instrumental convergence, mesa-optimization influences) are now mainstream in alignment.

Reached a mass audience. NYT bestseller, millions of podcast views, congressional testimony. If the goal is awareness, they've achieved more in 2 years of comms than in 20 years of research.

The governance team produces real work. The arXiv research agenda, treaty drafts, and international engagement are concrete contributions that other policy researchers cite as useful.

Weaknesses and Risks

Founder dependency and reputational fragility. MIRI's credibility depends overwhelmingly on Yudkowsky and Soares. Yudkowsky's more extreme statements (accepting nuclear war to prevent ASI, infanticide musings, "sex slave" posts) are well-documented and regularly surfaced by critics. If Yudkowsky's public image tips from "prophet" to "crank," MIRI's policy influence collapses.

No theory of political victory. MIRI advocates the most extreme policy position (global moratorium) without any credible pathway to its adoption. They have no political allies, no lobbying capacity, no grassroots organization. The governance team produces excellent analysis of what a moratorium could look like, but nobody is building the political coalition to enact one.

Financial sustainability crisis. $16M in reserves at $7-8M/year burn = ~2 years. The 2025 fundraiser fell $2.8M short. Open Phil stopped funding 5+ years ago. If the comms pivot doesn't attract new donors, MIRI faces existential financial pressure within 2-3 years.

Unfalsifiable reasoning. The Asterisk review's charge is devastating: MIRI's conclusions haven't changed despite the fundamental shift from hand-coded AI to deep learning. Their arguments were designed for a world of GOFAI and recursive self-improvement but are applied unchanged to a world of LLMs and scaling laws. If no evidence can change the conclusion, the reasoning isn't empirical -- it's ideological.

Alienation of the broader field. By dismissing the rest of AI safety as "alchemists" doing useless work, MIRI has burned bridges with the very people who might implement their ideas. When you need policy coalitions, having told everyone else they're delusional is a strategic liability.

Institutional culture concerns. The MIRI-CFAR ecosystem produced documented harms (psychotic episodes, cult dynamics). MIRI has not publicly addressed these. Anna Salamon, a MIRI board member, acknowledged creating "conditions for a cult." Even if MIRI itself was not directly responsible, the silence is a governance failure.

No independent governance. All five board members are insiders. For an org making sweeping claims about how the world should be governed, having no external accountability is a credibility problem.

Cross-References

Complementary to: PauseAI (shares goals but different tactics), MIRI provides intellectual underpinning that PauseAI's grassroots activism amplifies. Center for AI Safety (CAIS) shares the extinction risk framing but takes a more incremental approach.

In tension with: ARC (Paul Christiano's approach to alignment is exactly what MIRI considers futile). Anthropic (MIRI sees Anthropic as an accelerationist lab with safety branding). Redwood Research (empirical alignment work that MIRI considers too optimistic). OpenAI (MIRI sees OpenAI as the primary threat).

Contrasting model: METR/ARC Evals -- does practical evaluation work that MIRI might view as "making the best of a doomed situation" but that actually influences lab behavior today.

Gap-filling role: No other org says "stop everything" with MIRI's institutional weight. Whether this is valuable or counterproductive depends on whether you think the discourse benefits from having this extreme position represented.

What Would Change This Assessment

Upward revision if:

• AI systems produce a genuinely alarming capability jump (rapid self-improvement, escape attempts that succeed) -- this would validate MIRI's fast takeoff concern.
• A major AI incident (autonomous agent causing significant harm) creates political demand for MIRI's governance proposals.
• MIRI's governance team's treaty framework gets adopted by any government as a starting point for negotiations.
• MIRI successfully diversifies funding beyond Tallinn/SFF and individual EA donors.

Downward revision if:

• AI progress continues to follow smooth scaling laws for 2+ more years with no discontinuities.
• MIRI fails to secure sustainable funding and has to significantly cut operations.
• The political backlash against AI doomers intensifies (already underway) and MIRI becomes a liability to the broader safety movement.
• Yudkowsky's personal reputation deteriorates further due to controversial statements.

Self-Critique

What sources should I have checked but didn't? The full text of several key LessWrong posts (Death with Dignity, Paul-MIRI disagreement, Jessica Taylor's "My Experience at MIRI and CFAR") are on blocked domains. The search snippets were informative but I couldn't read them in full. The issarice timeline of MIRI would have been the most detailed chronological source.

Where is this analysis potentially biased? I may be giving too much weight to the "unfalsifiable reasoning" critique. It's possible that MIRI's core thesis is correct AND that their evidence base is weak -- these aren't mutually exclusive. A correct conclusion reached by flawed reasoning is still correct. Additionally, I may be anchoring too heavily on Yudkowsky's controversial statements when evaluating MIRI as an institution; the org is more than one person.

What would a thoughtful person who disagrees say? "You're evaluating MIRI by the standards of normal nonprofits when it's trying to do something unprecedented. Of course they don't have policy outcomes -- they're trying to prevent an extinction event that most people don't take seriously. The fact that the fundraiser fell short doesn't mean the mission is wrong, it means the world hasn't caught up yet. And your critique of their research era ignores that they were working on genuinely pre-paradigmatic problems where failure is expected."

What's my single weakest claim? That the comms/policy pivot is unlikely to produce results. It's only been 2 years. Historical precedents (nuclear non-proliferation, ozone layer, COVID response) show that paradigm-shifting policy can happen quickly once a crisis makes the case self-evident. If AI does produce a "warning shot" incident, MIRI's pre-positioned communications infrastructure could become enormously valuable.

What information would most change my view? A credible, detailed account of what MIRI's secret research actually produced (and why it failed). The nondisclosed-by-default policy means the most important part of MIRI's 10-year research era is invisible. If the work was much more substantial than the public-facing outputs suggest, the "research failure" story might be too harsh.