AI Impacts

Stated Theory of Change

AI Impacts claims that the most impactful work in AI safety is "understanding the situation" -- building the empirical evidence base about AI timelines, discontinuous progress, and risk scenarios -- rather than direct intervention (technical alignment, governance, policy). The mechanism is:

1. Researchers at AI Impacts investigate empirical sub-questions (e.g., how often do technologies see discontinuous jumps?)
2. These findings inform higher-level questions (e.g., should we expect a fast takeoff?)
3. Better answers to these questions guide everyone else's decisions: funders allocate better, alignment researchers choose better agendas, policymakers regulate more appropriately
4. The epistemic infrastructure (a publicly navigable wiki of modular conclusions) allows ongoing updating and disagreement-tracking

Katja Grace frames this as the analogue of climate science for AI risk. She argues that "understanding the situation" is 10-100x underinvested relative to intervention, and that "a person with a subjectively similar level of skill at everything under consideration will add more value via improving everyone's understanding of the situation by one person's worth of effort than they would by adding one person's worth of effort to pursuing the seemingly best intervention."

Revealed Theory of Change

What AI Impacts actually does, measured by its most-cited and most-influential outputs, is somewhat different from the stated theory:

What generates real-world influence: The Expert Survey (ESPAI). This is overwhelmingly what AI Impacts is known for and what propagates into media, Our World in Data, policy discussions, and community discourse. The survey is "understanding the situation" in the sense of measuring researcher sentiment, but it's closer to polling than to the "empirical science" Katja describes as the ideal.

The secondary high-impact output: The counterarguments and slowing-down posts. These are opinion/argumentation, not empirical research. They function more like intellectual advocacy than neutral science. The TIME op-ed is straightforwardly advocacy.

What gets much less attention: The wiki, the discontinuous progress investigation, the brain computation estimates, and the "tree of questions" research. These are the outputs closest to the stated theory of change ("empirical science about AI's future") but they generate far less impact than the survey or the blog posts.

The revealed theory of change is: (a) conduct a large, methodologically serious survey that serves as a shared reference point for the AI risk discussion, and (b) Katja Grace produces thoughtful, contrarian-leaning analysis that shifts the Overton window on specific topics (counterarguments, slowing down AI). The "wiki of empirical sub-questions" vision is a smaller fraction of actual output.

Key Assumptions

Assumption 1: Better understanding leads to better decisions.

• Evidence for: Plausible in principle. The survey has become a shared reference point that may have shifted the discourse. The counterarguments post sharpened the debate. Katja's slowing-down argument preceded the FLI letter.
• Evidence against: The FRI adversarial collaboration found that superforecasters and AI experts were "almost perfectly immune to evidence or arguments against their beliefs." If opinions are driven by "social and personality factors" rather than evidence, the entire theory of change is weakened.
• Testable: Partially. One could track whether specific AI Impacts findings were cited in specific policy decisions.
• If wrong: AI Impacts' work has aesthetic/intellectual value but limited causal impact on outcomes.

Assumption 2: Empirical sub-questions about AI are tractable and informative.

• Evidence for: The discontinuous progress investigation yielded a clear base rate. The survey methodology has been iteratively improved. Framing effects (HLMI vs FAOL) are a genuine discovery.
• Evidence against: The most impactful findings (5% median extinction risk, 2047 timeline) are "mood readings" of researchers, not empirical discoveries about AI itself. The tree of questions remains mostly unfilled after 10+ years.
• If wrong: The underlying research agenda is too ambitious for the resources available, and AI Impacts functions primarily as a polling organization.

Assumption 3: Understanding is 10-100x underinvested relative to intervention.

• Evidence for: In 2022, Katja estimated "between ten and a hundred times as much labor goes into intervening than understanding." The number of people doing empirical AI forecasting research is tiny.
• Evidence against: Epoch AI (much larger, better funded) now does quantitative AI forecasting. Academic groups have grown. The "understanding" space may no longer be as neglected as when Katja made this argument.
• If wrong: AI Impacts' marginal contribution is smaller than claimed, especially if Epoch AI covers the quantitative forecasting niche more rigorously.

Assumption 4: A small org can meaningfully contribute to this type of science.

• Evidence for: The survey is genuinely influential despite being produced by 2-7 people. The counterarguments post generated substantial community engagement from one person's work.
• Evidence against: The "army of climate scientists" comparison implies an ideal scale that AI Impacts has never approached and appears to be moving further from (7 staff in 2022, ~2 in 2025). The retention problems suggest the model may not scale.
• If wrong: AI Impacts is better understood as "Katja Grace plus occasional collaborators" than as an institution.

Strengths

1. The survey is a genuine public good. No one else has conducted a multi-year, methodologically serious, large-scale survey of AI researcher opinion. The methodology has been iteratively improved and is now peer-reviewed (JAIR). The findings provide a shared factual baseline for AI risk discussions. This is genuinely hard to replicate.

2. Intellectual independence from funders. The counterarguments post directly challenges the worldview of AI Impacts' funders. Katja's p(doom) of 7-19% is much lower than the median AI safety community member. This intellectual honesty makes the survey more credible and the org's outputs more trustworthy.

3. Katja Grace's thinking style. Her genuine uncertainty, willingness to steelman opposing views, and insistence on empirical grounding make her an unusually trustworthy voice. The podcast interviews reveal someone who thinks carefully rather than posturing. The TIME recognition reflects this.

4. Low overhead, high output per dollar. For ~$300-600K/year, AI Impacts produces the most-cited AI researcher survey in the world plus intellectual contributions that shift the discourse. The cost-effectiveness is remarkable compared to larger orgs.

5. Alumni pipeline. Past contributors include Paul Christiano (NIST), Daniel Kokotajlo (OpenAI whistleblower/AI Futures Project), and Zach Stein-Perlman (AI Lab Watch). The org appears to attract people who go on to high-impact roles.

Weaknesses and Risks

1. Single point of failure. AI Impacts is functionally "Katja Grace with support." If Katja becomes unavailable, there is no succession plan, no institutional knowledge base beyond the wiki, and likely no one to run the survey. The contraction from 7 to ~2 staff makes this worse.

2. Organizational fragility. No independent legal entity after 10+ years. No governance structure. No board. No annual reports. Operating under MIRI's 501(c)(3) creates opacity and dependency. The absence from SFF 2024-2025 without explanation raises questions about the org's trajectory.

3. The theory of change may not work. The FRI finding that "social and personality factors" determine AI risk opinions, not evidence, is a deep challenge. If the people AI Impacts is trying to inform are largely immune to updating on evidence, then even perfectly produced research has limited impact on decisions.

4. Epoch AI has eaten AI Impacts' niche. Epoch AI is larger, better funded, more quantitative, and covers compute trends, benchmarking, and forecasting that overlap with AI Impacts' stated mission. The distinctive contribution that remains is the survey and Katja's personal intellectual output.

5. The survey's most-cited findings may be misleading. The 2047 HLMI timeline is based on one of two framings; combining both gives 2073. The framing the public knows about is systematically shorter. AI Impacts has been transparent about this but the damage to public understanding is done.

6. No traceable policy impact. For an org whose theory of change is "understanding leads to better decisions," the inability to point to any specific decision that was improved by AI Impacts research is a weakness. The impact chain is: survey -> media/OWID -> public discourse -> ??? -> policy. The last links are unverified.

7. Retention problem never solved. The 2018 warning about retention proved correct. The 2022 expansion reversed within 2-3 years. The "rare skill set" problem (needing someone who is both a rigorous researcher and broadly knowledgeable about AI strategy) may be inherent and unsolvable at AI Impacts' salary levels.

Cross-References

• Epoch AI: The most direct comparison/competitor. Epoch is larger, more quantitative, and covers compute trends that AI Impacts also cares about. Complementary rather than redundant on the survey specifically. Adamczewski's reanalysis (he works at Epoch) suggests a collaborative relationship.
• MIRI: Fiscal sponsor and institutional home. Intellectually separate -- Katja's views differ markedly from Yudkowsky's. But the MIRI connection creates a perception problem that critics exploit.
• Open Philanthropy: Largest funder. The survey serves OP's need for information about AI timelines (Tom Davidson and Ajeya Cotra investigated AI Impacts grants). There's a circular dynamic: OP funds AI Impacts to produce information OP uses to make other funding decisions.
• FAR.AI: Where Rick Korzekwa went. Possible sign that the "do research" pathway through AI Impacts leads to more structured research environments.
• AI Lab Watch: Created by former AI Impacts researcher Zach Stein-Perlman. Shows how AI Impacts alumni spin off specialized projects.
• Metaculus/forecasting community: The survey provides a different signal from prediction markets -- expert sentiment rather than calibrated forecasts. Complementary.

What Would Change This Assessment

• Evidence that the survey changed a specific policy or funding decision would significantly strengthen the case for AI Impacts' impact. (Currently absent.)
• The 2024 survey results would help assess whether the org is still producing its core output.
• Clarification on the SFF absence (2024-2025) would help distinguish "funded through other channels" from "winding down."
• Evidence that Katja is actively recruiting or that the org is growing again would counter the contraction narrative.
• If the FRI finding replicated broadly -- that opinions on AI risk are immune to evidence -- this would significantly weaken the theory of change.
• If Epoch AI started running researcher surveys, AI Impacts' remaining distinctive contribution would shrink considerably.

Self-Critique

What sources should I have checked but didn't: LessWrong/Alignment Forum discussions of the counterarguments post and survey results (blocked domains). Academic citation analysis for all survey papers. Katja's recent (2025) blog posts on meteuphoric.com for any updates on org direction. Direct evidence about whether the 2024 survey was actually conducted.

Where is this analysis potentially biased: I may be too sympathetic to the "understanding over intervention" frame because it appeals to my analytical design. I may underweight the critique that AI Impacts' influence is primarily within the EA echo chamber and overweight its mainstream citations (Our World in Data, TIME).

What would a thoughtful person who disagrees say: "AI Impacts is a two-person project that produces one survey every year and some blog posts. Calling it a 'research organization' is generous. The survey's influence is largely within the x-risk community that funds it, creating a self-reinforcing cycle. The most interesting intellectual outputs (counterarguments, slowing down) are Katja's personal blog posts, not organizational products. The org has been unable to scale, retain staff, or achieve financial independence over a decade. It should either grow up or be honest that it's a personal project."

Single weakest claim: The claim that AI Impacts has had meaningful impact on decisions. The evidence chain stops at "widely cited in media and OWID" -- there is no evidence this translated into different decisions by policymakers, funders, or researchers.

What information would most change my view: Evidence that Katja is planning to shut down AI Impacts, or alternatively, evidence that she has secured new funding and is rebuilding the team. The current state of ambiguity makes it hard to assess whether this is a contracting project or a temporarily lean one.