Daniel Kokotajlo

Stated Theory of Change

Kokotajlo's stated theory of change has two layers:

Personal layer: Make the trajectory toward superintelligence legible to decision-makers who currently do not understand what is coming. The mechanism is scenario forecasting -- translating abstract timelines and capability curves into concrete narratives that policymakers, journalists, and the public can viscerally understand. The tabletop exercises extend this by making the experience interactive rather than passive.

Organizational layer (AI Futures Project): Produce a library of branching scenario forecasts, develop quantitative timelines and takeoff models, run tabletop exercises for policymakers, and eventually publish normative policy recommendations ("the policy playbook"). The long-term vision is a comprehensive scenario infrastructure that covers multiple timelines, political paths, and outcomes.

The causal chain is: scenario forecasting + TTX -> legibility of risk -> political will for regulation -> domestic regulation -> international coordination -> hardware verification + transparency -> slower/safer deployment -> time for alignment research to succeed.

Revealed Theory of Change

The revealed theory of change is somewhat broader than the stated one. Kokotajlo's actual activities suggest he operates a four-pronged strategy:

1. Information production (AI 2027, timelines models, AI 2030): Creating artifacts that make AI risk concrete
2. Information forcing (whistleblowing, Right to Warn, AIWPA): Creating legal/institutional channels for safety information to flow from labs to public
3. Legal/structural opposition (amicus brief, SB 1047 support): Using courts and legislation to constrain AI lab behavior
4. Community building (MATS mentorship, scenario contests, encouraging others to write scenarios): Scaling the approach beyond himself

The stated and revealed theories of change are closely aligned, which is unusual. The main divergence is that the stated theory emphasizes forecasting, while the revealed theory includes significant legal and political action that goes beyond pure information production.

One notable aspect: Kokotajlo is not trying to do alignment research himself. He is a philosopher and forecaster, not a machine learning researcher. His theory of change is explicitly about the political and informational prerequisites for alignment work to succeed, not the alignment work itself. This is a deliberate and defensible strategic choice.

Key Assumptions

Assumption 1: Timelines are short enough that urgency is warranted (confidence in assumption: medium) If AGI is 50+ years away, Kokotajlo's urgency-driven approach wastes resources that could be better deployed on slower institution-building. His own median (Dec 2030) makes this assumption plausible but far from certain -- he assigns meaningful probability (10-20%) to timelines beyond 10 years.

• Evidence for: METR horizon-length trend extrapolation, lab CEO statements, Metaculus/Manifold aggregate predictions
• Evidence against: Narayanan's diffusion barriers, history of AI winters, GPT-5 as "on trend but not a big deal"
• Testable: Yes, through METR trend, near-term AI 2027 predictions, economic productivity data

Assumption 2: Making risk legible changes behavior (confidence: medium-low) This is the deepest assumption. Even if everyone understands the risk, it does not follow that behavior changes. Marcus's counterargument -- that AI 2027 actually accelerates the race by serving as marketing material -- directly challenges this. The track record is mixed: Right to Warn -> AIWPA is a clear success; but the broader policy environment (deregulation, AI race rhetoric) has not shifted toward coordination.

• Evidence for: Right to Warn -> AIWPA pipeline, JD Vance reading AI 2027
• Evidence against: No meaningful US AI regulation, continued race dynamics, OpenAI proceeding with for-profit conversion
• What changes if wrong: Kokotajlo's entire approach loses its foundation. If information production does not change behavior, he needs a different lever.

Assumption 3: International coordination is achievable (confidence: low) Kokotajlo's ideal outcome requires US-China cooperation on AI governance, including mutual hardware verification. The current geopolitical reality offers little support for this. His own tabletop exercises show that most games end in race dynamics, not coordination.

• Evidence for: Historical precedent (nuclear arms control), demand for deals in TTX games
• Evidence against: Current US-China relations, Taiwan tensions, no existing institutional framework
• Testable: Partially, through diplomatic developments over next 2-3 years

Assumption 4: The "software-only singularity" can be slowed by political action (confidence: medium) If the intelligence explosion happens entirely within a few data centers in a few months, political action is too slow to intervene. Kokotajlo's theory of change requires either (a) enough warning time for politics to matter, or (b) sufficient regulation already in place when the explosion begins. The software-only singularity described in AI 2027 leaves very little time for political response.

Strengths

Credibility from sacrifice: The $2M equity risk is among the strongest credibility signals in AI safety. Unlike most commentators, Kokotajlo has paid a personal price for his views.

Forecasting track record: "What 2026 Looks Like" is a documented case of accurate directional prediction on specific technical milestones, written before ChatGPT. This distinguishes him from the many people who make vague predictions about AI.

Intellectual honesty under pressure: His willingness to publicly lengthen timelines (2027 -> 2030), acknowledge model errors, pay bounties for criticism, and express genuine uncertainty (30-40% on current paradigm plateauing) sets him apart from advocates on both sides.

Concrete policy vision: Unlike many AI safety advocates who stop at "we need regulation," Kokotajlo names specific mechanisms (hardware verification, chip tracking, transparency requirements) and specific use cases (mutual inspection for international treaties).

Demonstrated real-world impact: The Right to Warn -> AIWPA pipeline is the clearest causal chain from AI safety advocacy to concrete legislation that I have seen in any org analysis.

Reach beyond the bubble: AI 2027 reached 1M+ readers and was read by a sitting US Vice President. The TTX program has engaged congressional staffers and journalists, not just AI researchers. This is unusually broad for AI safety work.

Weaknesses and Risks

Counterproductivity risk: Marcus's argument that AI 2027 serves as "marketing materials" for AI labs is the most serious challenge. If the net effect of making superintelligence seem imminent is to increase investment in racing rather than in safety, Kokotajlo's work is counterproductive regardless of its accuracy. He acknowledges this concern but cannot resolve it empirically.

Scenario forecasting is fundamentally a single-point estimate: Despite disclaimers about branching futures, the narrative format of AI 2027 creates a psychological sense of inevitability. MIRI's critique about the "planning fallacy" in sequential scenario construction is apt -- if you imagine the most plausible next step at each point, you systematically underestimate delays.

AIFP is fragile: A 5-person team with opaque funding, no visible board, and one public-facing figure creates extreme key-person risk. If Kokotajlo's credibility were damaged (e.g., by a dramatically wrong near-term prediction), AIFP has no organizational resilience.

The gap between reaching policymakers and changing policy: Having JD Vance read AI 2027 is not the same as JD Vance acting on it. The US policy landscape remains dominated by deregulation and AI race rhetoric. There is a risk that Kokotajlo's work creates the appearance of policy engagement without the substance.

Narayanan's deployment/diffusion challenge: Even if AI capabilities advance rapidly, the "long causal chain between capability and societal impact" means that the world might look much more normal than AI 2027 predicts for much longer than expected. Narayanan conceded Kokotajlo's tech predictions were accurate but his social impact predictions were not. If social impacts continue to lag capabilities, the political urgency Kokotajlo needs may never materialize.

Echo chamber risk: AIFP was built by Lightcone, reviewed by EA/rationalist community members, and operates within the AI safety ecosystem. Despite the 60+ reviewers, the core worldview originates from a relatively narrow intellectual community. The Hacker News reaction shows how the broader tech community receives this work.

Cross-References

Complementary to: MIRI (shares worldview, different focus), Anthropic's RSP framework (AIFP advocates for lab commitments), Center for AI Safety (different approach to same goal), PauseAI (more radical version of similar theory of change).

In tension with: Narayanan / AI Snake Oil (fundamentally different causal model), mainstream AI policy establishment (which focuses on near-term harms rather than existential risk), AI lab leadership (which claims to manage risk internally).

Overlaps with: The Midas Project (accountability/transparency advocacy), Center for AI Policy (policy advocacy, Thomas Larsen connection), METR (uses their evaluations as key input data).

Unique positioning: Kokotajlo occupies a genuinely unique position as a former lab insider with a documented forecasting track record who is now an independent advocate. There is no one else doing exactly this combination of scenario forecasting + whistleblowing + TTX + policy advocacy.

What Would Change This Assessment

• Near-term predictions proving dramatically wrong: If AI 2027's 2025-2026 predictions are clearly inaccurate by mid-2026, his forecasting credibility declines sharply.
• A major policy win attributable to his work: If AIWPA passes and is used, or if hardware verification funding increases due to his advocacy, the theory of change strengthens.
• AI progress clearly plateauing: If the METR horizon-length trend bends downward, the urgency case weakens substantially.
• AIFP funding transparency: Learning who funds AIFP would allow assessment of financial independence.
• Lab behavior changing in response to AI 2027: If major labs cite AI 2027 scenarios in their safety cases or modify behavior based on TTX outcomes, the information-production theory of change is validated.
• Counterproductivity evidence: If investment in AI racing demonstrably increased because of AI 2027 framing (hard to measure), the Marcus critique would be validated.

Self-Critique

Limitations of this analysis:

• I could not access Kokotajlo's own LessWrong/AF posts directly (hundreds of posts/comments that constitute primary sources for his thinking). I relied on secondary accounts and the Grokipedia article.
• AIFP's funding is completely opaque. My positive assessment of incentive alignment could be wrong if there are undisclosed funders with agendas.
• I may be giving too much weight to the equity sacrifice as a credibility signal. It is genuinely unusual, but it happened two years ago, and the money was ultimately retained.
• The counterproductivity question is fundamentally unresolvable with available evidence. I have no way to measure whether AI 2027's net effect on the AI race is positive or negative.
• I may be underweighting the "echo chamber" concern. The 60+ reviewers of AI 2027 are mostly drawn from the EA/rationalist community, and the positive reception may reflect in-group validation rather than genuine external credibility.

What a thoughtful disagreer would say: "Kokotajlo is a talented scenario writer who has leveraged a single accurate prediction and a dramatic resignation into outsized influence. AI 2027 is sophisticated science fiction, not science. The real damage is not that his predictions are wrong -- it is that his predictions normalize the idea that superintelligence is imminent, which accelerates the very race he claims to oppose. The policy impact (AIWPA) is real but modest. The TTX program preaches to the converted. And the entire enterprise depends on short timelines being correct -- if they are not, the opportunity cost of the attention and resources directed toward Kokotajlo's urgent framing is enormous."

My weakest claim: That Kokotajlo's work has net positive impact on AI safety outcomes. The counterproductivity argument is strong enough that I cannot rule out net negative impact.

Information that would most change my view: A systematic assessment of AI 2027's near-term predictions against reality, combined with evidence about whether AI investment or AI safety investment increased more in response to the publication.