Existential Risk Observatory

Stated Theory of Change

ERO claims that public awareness of existential risk is the necessary first step toward government action. The mechanism:

1. AI researchers identify and publish x-risk concerns
2. Journalists and public communicators translate these for broader audiences
3. Activists pressure governments to act
4. Governments implement risk-reduction policies

ERO positions itself primarily at step 2 (researcher-to-journalist translation) while also contributing to step 4 via specific policy proposals. The Conditional AI Safety Treaty is the culmination of this -- it names the specific policy instrument ERO believes governments should adopt.

The key assumption is that a critical mass of public awareness will create political demand for x-risk regulation. Otto estimates current US awareness at ~24% (up from 7% in late 2022) and implicitly believes a threshold around 50%+ would enable meaningful regulation.

Revealed Theory of Change

ERO's actions are largely consistent with its stated theory, which is unusual and worth noting. The org says it does media advocacy and awareness research, and that is indeed what it does. There is no significant gap between stated and revealed priorities.

However, there are subtle differences:

Stated emphasis: "informing the public debate" -- This sounds neutral and educational. Revealed emphasis: persuasion. The message testing research is explicitly about finding "the most effective ways" to communicate x-risk, testing "which narratives are most appealing to specific groups," and optimizing "conversion rates." This is advocacy, not education. ERO occasionally uses the language of neutral information provision when the actual work is persuasion. This is standard for advocacy orgs but worth naming.

Stated: broad x-risk focus (AI, pandemics, nuclear, climate). Revealed: AI is 90%+ of the output. The website has sections on all x-risks, but virtually all media output, events, and research focus on AI. The "existential risk observatory" name is broader than the actual work, which is more accurately described as "AI x-risk communications."

Stated: the treaty paper positions ERO as a policy research org. Revealed: the paper is competent but derivative. It reviews existing proposals rather than generating novel research. ERO's comparative advantage is not in policy research -- it's in media placement and public communication.

Key Assumptions

Assumption 1: Public awareness drives policy action on x-risk.

• Evidence for: Climate change analogy -- decades of awareness-building preceded policy action. The FLI open letter (2023) demonstrated that public attention can shift the Overton window. PNAS paper (2025) found x-risk narratives "do not distract" from other AI concerns.
• Evidence against: Public concern about nuclear weapons has been high for decades without leading to disarmament. Awareness of pandemic risk was high before COVID but didn't prevent preparedness failures. AI policy is currently driven more by industry lobbying and great-power competition than public opinion.
• Testable: Yes. Track correlation between awareness survey numbers and AI safety policy adoption over time.
• If wrong: ERO's entire approach fails. If policy is determined by industry lobbying and geopolitics regardless of public opinion, then awareness-building is low-impact.

Assumption 2: Media placement is the most effective awareness-building tool.

• Evidence for: ERO's own surveys show media exposure increases concern. TIME has 60M+ monthly readers. The four TIME op-eds represent enormous reach per dollar.
• Evidence against: Op-ed impact is fleeting. Michael Dickens notes media outreach is "not my favorite thing" for impact. The AI safety community increasingly emphasizes direct political engagement (ControlAI, PauseAI US lobbying) over media.
• If wrong: ERO should pivot toward direct government engagement, which it has partially done with policy proposals but without the same institutional footprint as ControlAI or ARI.

Assumption 3: A Netherlands-based org can influence global AI policy.

• Evidence for: Four TIME op-eds demonstrate international media reach. The Paris debate with Bengio shows international convening power. EU policy is made partly in Brussels, accessible from Amsterdam. The Netherlands has influence in EU AI Act discussions.
• Evidence against: The US and China are the two countries that matter most for AI development. ERO has no Washington or Beijing presence. Dutch policy has minimal direct influence on frontier AI development. Dickens considers this ERO's primary weakness.
• If wrong: ERO's domestic advocacy (Dutch Parliament, ANBI petitions) has limited global relevance. International media work partially compensates but lacks the sustained relationship-building that effective policy influence requires.

Assumption 4: A conditional treaty is politically feasible.

• Evidence for: Precedent in nuclear nonproliferation, chemical weapons, climate treaties. The "conditional" framing avoids the political toxicity of an unconditional pause. AISIs already exist as potential enforcement bodies.
• Evidence against: US-China coordination on AI is extremely challenging in the current geopolitical environment. The US has deregulated AI under the Trump administration. China has no incentive to accept international oversight of its AI development. No country has signed anything resembling this treaty.
• If wrong: The treaty proposal remains an intellectual exercise. The value then depends on whether proposing the treaty shifts the Overton window even if the treaty itself is never signed.

Strengths

1. Media access punch above their weight. Four TIME op-eds from a sub-$300K Dutch foundation is extraordinary. This is ERO's clearest comparative advantage -- Otto can get published in outlets that most AI safety orgs cannot access.

2. Otto's measured tone. In a space dominated by either extreme alarm (Yudkowsky) or industry optimism (Lecun), Otto occupies a credible middle ground. His "luck" framing is original and resonant: "My pdoom is not 99%, but it's an appreciable figure. And what bothers me, is that it seems to be a game of luck." This is effective for persuading people who aren't already converted.

3. Theory of change evolution shows learning. The shift from generic awareness (2021) to specific policy proposals (2024-2025) demonstrates that ERO has updated its approach based on experience. The Conditional AI Safety Treaty is a more tractable policy target than "AI Pause."

4. Bengio relationship. Getting a Turing Award winner to repeatedly participate in ERO events is a significant endorsement of credibility. This is a genuine asset that larger, better-funded orgs would envy.

5. Independence from AI labs and the dominant funder. ERO has no economic ties to AI labs and no Coefficient Giving funding. This gives it complete freedom to criticize industry. In a field where many safety orgs have complex financial relationships with labs, ERO's independence is valuable.

Weaknesses and Risks

1. No evidence of policy impact. ERO has submitted 14+ policy proposals but there is no documented case where ERO's advocacy led to a specific policy change. The gap between media reach and policy influence is the central question about ERO's effectiveness. Media placement is a means, not an end, and the causal chain from op-ed to policy is unproven.

2. Financial opacity and precariousness. Budget is unknown and likely small ($100K-$300K). Funded by essentially two individuals (Tallinn, Schuurman). Self-described as "funding-constrained." If either funder withdraws, ERO faces an existential crisis. The lack of accessible financial data undermines credibility for an org advocating transparency in AI development.

3. Thin organizational capacity. 15+ people listed but mostly volunteers/interns. No one on the team has frontier AI research experience. Only one person has an AI degree. The team is heavily humanities/social science, which is appropriate for communications but insufficient for technically credible policy research.

4. Netherlands location limits influence on the two most important actors (US, China). Despite international media reach, ERO has no sustained presence in Washington, London, Beijing, or Brussels. Policy influence requires sustained relationship-building, not just op-eds.

5. The message testing criticism has legs. While Weiss-Blatt's Cambridge Analytica comparison is overblown, the underlying point -- that ERO/CfAIS approaches public communication as a marketing problem (demographic targeting, narrative optimization, conversion rates) -- is a genuine reputational risk. If ERO is perceived as "propagandists" rather than "educators," its credibility as a neutral information source is undermined.

6. Governance concerns. Two-person governance (Director + Treasurer), no visible independent board oversight, ANBI financials on a personal Google Drive. For an org seeking to influence public policy, this level of governance is a weakness.

Cross-References

• PauseAI: ERO has collaborated with PauseAI on events and protests. PauseAI is more grassroots, does direct action, and has a simpler message ("pause AI"). ERO is more polished and policy-oriented. They are complementary rather than competing -- ERO provides intellectual legitimacy, PauseAI provides street presence.

• ControlAI: More professionalized and better-funded than ERO. Operates in UK and expanding to US. Pursues "inside game" strategy (briefing lawmakers). ControlAI is what ERO might become with 10x the budget and a London/DC presence.

• MIRI: MIRI's post-2024 pivot to policy and public outreach puts it partly in ERO's territory but with vastly more resources and brand recognition. MIRI's book and comms team are more impactful than ERO's media work.

• Future of Life Institute: FLI operates in a similar space (media + policy) but with far more resources and a Boston/Brussels presence. ERO co-organized events with FLI staff and published joint op-eds in Dutch media.

• Centre for Future Generations / ICFG: Steven Schuurman's Brussels think tank, which also focuses on existential risk governance. Sam Bogerd (ERO former volunteer) moved to CFG. There's a Schuurman-funded ecosystem: ICFG/CFG for Brussels policy, ERO for Dutch/international media.

What Would Change This Assessment

• Evidence of policy influence: If ERO could demonstrate that a specific government action (e.g., a Dutch amendment to the EU AI Act, inclusion of AI x-risk in a national risk assessment) resulted from their advocacy, this would significantly upgrade the assessment.
• Budget growth to $1M+: With meaningful budget, ERO could hire full-time policy staff, establish a DC or Brussels presence, and move from op-ed writers to policy players.
• Academic citation of treaty paper: If the arXiv paper gets cited in policy documents or academic literature, it validates ERO's policy research capability.
• Coefficient Giving funding: Would signal that the largest AI safety funder considers ERO effective.
• Failure to maintain media access: If TIME and similar outlets stop publishing ERO's work, the org's primary asset is gone.
• Funder withdrawal: If Tallinn or Schuurman stop funding, ERO likely cannot survive.

Self-Critique

What sources should I have checked but couldn't:

• The ANBI financial PDF on Google Drive (non-extractable)
• Dutch-language media coverage (paywalled)
• London Futurists podcast with Otto (audio only, no transcript)
• The full policy proposals PDF (binary, not extractable)
• Any internal communications or grant applications that would reveal ERO's actual strategy documents

Where this analysis is potentially biased:

• I may underweight ERO's domestic Dutch impact because all Dutch-language evidence was inaccessible. ERO's Netherlands advocacy could be significantly more influential than what English-language sources reveal.
• I may overweight the Weiss-Blatt critique because it is the most detailed critical source. There may be defenses of ERO's approach that I didn't find.
• The "too small to matter" framing may undercount the value of seeding ideas. The Conditional AI Safety Treaty could become influential even if ERO itself remains small.

What a thoughtful person who disagrees would say: "You're evaluating ERO by whether it has changed policy directly, but that's the wrong metric for an awareness-building org in year 4. ERO's four TIME op-eds have reached tens of millions of people. You can't trace individual policy changes to media coverage, but that doesn't mean the coverage didn't matter. ERO is doing exactly what it should be doing: building the intellectual foundation and public awareness that will enable policy change when a crisis moment arrives. And with a $44K SFF grant, the cost-effectiveness per unit of media reach is extraordinary."

My single weakest claim: That ERO's Conditional AI Safety Treaty is "derivative." It synthesizes existing proposals but the specific combination (conditional trigger, AISI delegation, no exemptions for international research institutes) may be novel and valuable. I may be underrating the contribution of competent synthesis in a field full of fragmentary proposals.

What information would most change my view: Accessible financial data showing ERO's actual budget, FTE count, and cost-per-output. If ERO is producing this volume of output on $150K/year with 3 paid staff, the cost-effectiveness argument becomes very strong. If the budget is $500K+ with 10+ paid staff, the output level is less impressive.