Simon Institute for Longterm Governance

Stated Theory of Change

SI's stated theory of change is that the binding constraint on international AI governance is not lack of knowledge or political will, but lack of translation between technical AI communities and the multilateral diplomatic system. Their mechanism:

1. Bridge the inferential gap between AI researchers (who understand the technology but not institutions) and diplomats (who understand institutions but not the technology)
2. Build shared understanding through training, facilitation, and institutional design -- not through producing their own policy recommendations
3. Design multilateral institutions (the Scientific Panel, Global Dialogue) that can produce ongoing, legitimized risk assessments
4. These assessments create the foundation for coordinated governance action

The core bet: patience and usefulness beat urgency and alarm. Konrad articulates this explicitly: diplomats already feel maximum urgency on their existing priorities, so the only way in is to make yourself useful within their paradigm, not to impose an external one.

Revealed Theory of Change

SI's actions largely align with their stated theory. Evidence of alignment:

• They do not produce policy recommendations (stated and observed)
• They invest heavily in diplomat training and relationship-building (125+ diplomats, 800+ calls, 50+ missions in 2024)
• They co-author papers with established institutions (Carnegie, Oxford Martin) rather than publishing unilaterally
• They hire people with genuine UN system experience (Renata Dwan)
• They adapted messaging for different audiences and were transparent about this

Where actions diverge slightly from the stated theory:

• US-China mediation ambitions (mentioned in Luma event, Konrad's personal site) go well beyond the public-facing "facilitation" framing. This is more like track-two diplomacy -- a higher-risk, higher-impact activity than training courses.
• The Bay Area outreach (2025-2026) suggests SI is starting to engage directly with AI labs, not just governments. This is new and represents a theory-of-change expansion.
• Benefit-sharing work (TechPolicy.Press article, Windfall Trust workshop) pushes into economic redistribution territory that goes beyond risk assessment. This may reflect influence from the Global South engagement mandate.

Key Assumptions

Assumption 1: Shared scientific understanding is a necessary precondition for coordinated governance action.

• Evidence for: IPCC reports did eventually drive climate policy (Paris Agreement). The Carnegie paper makes this case persuasively.
• Evidence against: The IPCC took 30+ years from founding (1988) to meaningful binding commitments (2015). AI timelines may be much shorter than 30 years. Also, the US withdrew from the Paris Agreement (twice), showing that shared understanding does not guarantee cooperation.
• Testable: If the Panel's first report (July 2026) is cited in national AI policies within 2 years, this assumption is supported.
• If wrong: SI's institutional design work is well-crafted but irrelevant -- the institutions produce reports nobody acts on.

Assumption 2: The UN system can govern AI meaningfully despite the US withdrawal.

• Evidence for: China, EU, and 190+ member states still support the process. The IPCC functioned despite periodic US hostility. Norms can influence behavior even without the leading power.
• Evidence against: The US controls the most advanced AI labs. Without US buy-in, the Panel's risk assessments are about systems it cannot examine, from companies that have no obligation to engage. China's enthusiastic support may discredit the process in US eyes.
• Testable: If any major AI lab provides data or cooperation to the Panel by 2027, this assumption gains support.
• If wrong: The Scientific Panel becomes a legitimate but powerless institution, producing reports that are diplomatically important but technically limited.

Assumption 3: Facilitation-based work can be done effectively by a 10-person org.

• Evidence for: SI's track record shows outsized output relative to budget. The Year One Review documents significant activity on CHF 144K. The "sole org" claim for frontier AI multilateral governance appears true.
• Evidence against: A single grant rejection or political shift could end the organization. Key-person risk is high (Konrad and Max are irreplaceable for the network).
• Testable: If SI maintains operations and influence through the 2026-2028 period with staff growth to 15+, this is validated.
• If wrong: SI is a fragile institution where the founders' personal networks, rather than organizational capacity, are the actual asset.

Assumption 4: Training diplomats on AI leads to better governance outcomes.

• Evidence for: 9.1/10 training rating, repeated invitations from permanent missions, diplomats requesting follow-up sessions. The GDC negotiations reportedly benefited from SI-trained diplomats.
• Evidence against: Training provides awareness but not authority. The diplomats who attend SI trainings may not be the ones who make decisions. Domestic politics, not diplomatic knowledge, drives most countries' AI governance positions.
• If wrong: SI provides a popular educational service that feels impactful but does not change the decisions that matter.

Strengths

Unique niche with genuine demand. No other organization focuses exclusively on frontier AI governance at the multilateral level. GovAI, IAPS, CSET focus on US/UK/EU domestic policy. This is a real gap, and SI fills it. The fact that UN bodies commission SI for reports and permanent missions invite them for trainings validates the demand.

Lean and cost-effective. ~$730K/year for an org that trained 125+ diplomats, published institutional design research, maintained 800+ calls, and contributed to a UNGA resolution. By comparison, a single senior researcher at a US think tank costs more than SI's entire budget.

Renata Dwan hire. Bringing on the former UNIDIR Director and Tech Envoy adviser gives SI genuine insider access. This is not a symbolic advisory board appointment -- she is a full-time Director of Tech Diplomacy.

Honest self-assessment. SI's Year One Review is unusually transparent about failures (SFF rejection, minimal EA Funds support, limited track record). Their explicit acknowledgment of downside risks and mitigation strategies is rare among EA-adjacent orgs.

Concrete institutional output. The Scientific Panel adoption (Aug 2025) is a tangible deliverable. Whether SI's contribution was decisive is debatable, but their institutional design work was clearly part of the process.

Weaknesses and Risks

US withdrawal is devastating for the theory of change. The most advanced AI development happens in the US. The Trump administration explicitly rejects multilateral AI governance and calls existential risk an "ideological fixation." If the US continues this posture through 2028+, the institutions SI helped build become marginal. China's support makes this worse, not better, from a US perspective.

No enforcement mechanism. The Scientific Panel assesses risk; it does not regulate. Even the IPCC, after decades of excellent reports, could not prevent global emissions from rising. The missing step from "shared understanding" to "changed behavior" is SI's biggest theoretical gap, and they have not articulated how to close it.

Extreme funder concentration. One Open Phil grant (~$730K/year) likely represents the majority of SI's budget. This creates existential risk: if Open Phil's strategy shifts, or if they evaluate SI's impact as insufficient, the organization could collapse. The early funding history (SFF rejected, EA Funds gave 1/12th) shows how precarious this landscape is.

Key-person risk. Konrad and Max are the network. Their personal relationships with diplomats, researchers, and policymakers are SI's core asset. This is inherent in facilitation-based work but makes the organization fragile.

Unmeasurable impact. Facilitation is inherently hard to evaluate. SI's counterfactual impact claims are "educated guesses" (their words). After 5 years, no independent evaluation exists. The difficulty of measuring their impact makes it hard for funders to assess value-for-money and hard for SI to prove they deserve continued support.

Two departures from small team. Belinda Cleeland (Director of Policy) and Sofia Mikton left without public explanation. In a 10-person org, losing two staff -- including the policy lead -- is significant. Whether this reflects normal turnover or internal issues is unknown.

Cross-References

Complementary to: GovAI, IAPS, CSET (domestic AI policy -- different level of governance). CSER Cambridge (SI's early collaborator on science-policy interface). Future of Life Institute (co-organized events; FLI focuses on advocacy, SI on facilitation). Longview Philanthropy (potential funder, not competitor).

Compared to: The closest analogue is CERN's founding story, which Konrad explicitly draws on. SI sees itself as the "Amaldi" figure -- the person who builds the institution rather than getting the credit. The CERN analogy is imperfect (CERN had US defense backing; SI does not have equivalent great-power support), but the diplomatic entrepreneurship pattern is similar.

Gap they fill: No other AI safety org has deep, ongoing engagement with the UN system, permanent missions in Geneva and NYC, and relationships with Global South diplomats. This is genuinely unique.

Potential redundancy: If the UK AISI or OECD successfully builds a multilateral AI risk assessment process independent of the UN, SI's value diminishes. But the Carnegie paper (co-authored by SI) explicitly argues for complementary processes, not replacement.

What Would Change This Assessment

• US re-engagement with multilateral AI governance would dramatically increase the value of SI's work. A change in US administration or a bipartisan AI safety event could shift this.
• Scientific Panel producing a consequential first report (July 2026) that is cited by national governments would validate the institutional design.
• Open Phil non-renewal would test whether SI can survive. If they secure diversified funding, the organization is more robust than it appears.
• AI lab engagement with the Panel (providing data, cooperating with assessments) would bridge the gap between assessment and influence.
• A major AI incident that creates global demand for coordinated governance would make SI's existing institutional infrastructure extremely valuable.
• Evidence that trained diplomats made different decisions as a result of SI training would strengthen the case for facilitation-based theory of change.

Self-Critique

Weakest claim: That SI's contribution to the Scientific Panel design was meaningfully counterfactual. 190+ member states negotiated; many other institutions contributed design ideas. I cannot distinguish SI's marginal impact from the broader policy process.

Potential bias: I may be overweighting SI's self-reported impact because their internal communications are unusually candid and intellectually honest. Candor does not equal accuracy. The Year One Review reads as impressively self-aware, but it is still written by the founders for a donor audience.

What I missed: I could not access SI's annual financial reports (PDF on Google Drive), their full board composition (Notion page), or the US-China rapprochement memo (Google Doc). These are real gaps. The financial reports in particular might reveal concentration or dependencies not visible from the aggregate numbers.

What a thoughtful critic would say: "SI is building sand castles against a rising tide. The institutions they design are technically well-crafted, but the geopolitical reality is that the US -- which controls the labs that matter -- has rejected multilateral governance. China's enthusiasm for the UN process is not a sign of shared commitment to AI safety; it is a geopolitical strategy. SI's patient, facilitation-based approach is admirable in peacetime, but we may not have the decades that IPCC-style institution-building requires."

What would most change my view: (1) Evidence that the US re-engages with multilateral AI governance within 2 years. (2) Evidence that the Scientific Panel's first report directly influences a national government's AI policy. (3) An independent evaluation of SI's impact by a credible external assessor.