SaferAI

Stated Theory of Change

SaferAI states a clear three-part mechanism for reducing AI risk:

1. Measure: Develop quantitative risk assessment tools that translate benchmark capabilities into real-world risk estimates, using methodologies adapted from aviation, nuclear, and financial risk management.
2. Rate: Publicly score frontier AI companies' risk management practices against these tools, creating reputational pressure and providing data for regulators, investors, and the public.
3. Standardize: Embed SaferAI's risk management philosophy into international standards (EU AI Act, CEN-CENELEC, ISO, OECD, NIST) so that regulatory infrastructure institutionalizes aviation-style safety requirements for AI.

The underlying assumption is that alignment tools will eventually exist, but labs won't implement them without external pressure. Simeon Campos framed it explicitly: SaferAI is the enforcement mechanism for alignment work done by others. The causal chain: independent measurement -> public accountability -> regulatory standards -> mandatory implementation -> reduced AI risk.

Revealed Theory of Change

SaferAI's actions mostly align with its stated theory, with some notable divergences:

Consistent with stated theory:

• The ratings product exists and has been released twice with expanding coverage (6 companies, then 12)
• Standards body participation is extensive and deep (writing standards, not just commenting)
• The EU CBRN contract represents the theory working: SaferAI's risk modeling expertise is being directly used by regulators
• The FLI Safety Index collaboration multiplies SaferAI's reach through a more established partner
• The quantitative risk modeling papers provide the technical foundation for the entire project

Divergences from stated theory:

• The stated emphasis on "independence" is complicated by the advisory business. Advising G42 on their framework and then rating it reveals a consulting-and-rating dual role that the stated theory doesn't acknowledge.
• The commercial revenue development (Gabor Szorad) suggests SaferAI may be moving toward a model where companies PAY for advice and then get RATED by the same organization. This hasn't happened yet, but the trajectory points there.
• The financial opacity contradicts the transparency principles SaferAI applies to others. If the theory of change depends on credibility as an independent assessor, and credibility depends on demonstrated independence, then hiding your funder list undermines the mechanism.

Key Assumptions

Assumption 1: Public-information-based ratings create meaningful pressure on companies.

• Evidence for: xAI improved from 0% to 18%, Meta from 14% to 22% between the two rating rounds. TIME, NBC, Fortune cover the ratings. Yoshua Bengio endorses them.
• Evidence against: Anthropic and Google declined. Companies can and do dismiss the ratings as incomplete ("our comprehensive approach extends well beyond what's captured" -- Google). No evidence that any company changed practices specifically because of SaferAI ratings vs. other pressures.
• Testable: Track whether companies' scores improve over multiple rounds and whether improvements correlate with rating-specific concerns.
• If wrong: SaferAI becomes a scorekeeper that no one listens to -- all measurement, no influence.

Assumption 2: Aviation/nuclear risk management frameworks can be meaningfully adapted to AI.

• Evidence for: The risk management literature is mature and industry-agnostic at the framework level. The specific adaptations SaferAI proposes (KRI/KCI thresholds, risk modeling, Delphi studies) are concrete. The EU Code of Practice has adopted several elements SaferAI advocated for.
• Evidence against: Stuart Russell: "none of the current activity provides any kind of quantitative guarantee of safety; nor does it seem possible to provide such guarantees given the current approach to AI via giant black boxes." AI systems change fundamentally with each new generation in ways that planes and nuclear reactors don't. The threat landscape is adversarial in a way that aviation isn't.
• Testable: Whether the quantitative risk models produce estimates that correlate with actual incidents/harms as capabilities advance.
• If wrong: SaferAI is importing a methodology from a domain where it worked into a domain where it may be fundamentally inapplicable.

Assumption 3: Standards bodies are the right lever for influencing AI development.

• Evidence for: The EU AI Act is the most comprehensive AI regulation globally. Standards bodies shape what compliance looks like. SaferAI has people literally writing the standards. Historical precedent: automotive safety standards transformed the industry.
• Evidence against: The US has rolled back AI oversight under the current administration. Chinese companies are outside the EU's regulatory reach. 46 tech CEOs sent a letter asking for a two-year pause on EU GPAI obligations. Enforcement is uncertain.
• Testable: Whether EU standards propagate globally or remain regional. Whether enforcement actions follow from the standards SaferAI helped write.
• If wrong: SaferAI has invested heavily in a European regulatory framework that may be geopolitically contained while the most important AI development happens in the US and China.

Assumption 4: SaferAI can maintain independence while developing commercial relationships with the companies it rates.

• Evidence for: The Anthropic downgrade while Fabien Roger sits on the board. The right-of-reply process is transparent.
• Evidence against: The G42 advisory/rating dual role. The trajectory toward commercial advisory revenue. The absence of a published conflict of interest policy.
• Testable: Whether SaferAI publishes a COI policy, discloses all advisory relationships, and shows consistent willingness to rate clients poorly.
• If wrong: SaferAI becomes captured by the industry it's supposed to assess -- the exact failure mode that the 2008 financial crisis revealed in credit rating agencies.

Strengths

1. Unusually concrete and measurable theory of change. Most AI safety orgs have vague theories of change ("do alignment research" or "influence the policy conversation"). SaferAI's three-pronged approach -- measure, rate, standardize -- has clear intermediate steps and observable outputs. You can point to the ratings, the standards contributions, and the EU contract as evidence of the mechanism working.

2. Strong team for the specific niche. A Superforecaster running risk modeling, a spacecraft engineer writing standards, an ex-OECD policy head advising on governance, an ex-Berkeley researcher running operations. This is not a typical EA org; the team was deliberately assembled from industries with risk management cultures.

3. Media reach disproportionate to size. Three TIME features, a TIME op-ed, IEEE Spectrum, NBC, Fortune -- for a ~15-person French nonprofit. This amplifies the ratings' accountability pressure well beyond SaferAI's direct reach.

4. EU CBRN contract is a genuine inflection point. Government contract revenue, multi-year stability, direct influence on EU regulatory enforcement. This is the theory of change actually working: SaferAI's expertise is being purchased by the people who will enforce AI regulation.

5. Standards body access is a structural advantage. People literally writing the standards that will shape regulation. This is slow, unsexy, high-leverage work that few AI safety orgs pursue.

6. Novel technical contributions. The LLM-simulated expert elicitation approach is genuinely innovative. If validated, it makes quantitative risk assessment dramatically more scalable. The cyber risk models are the most detailed proof-of-concept available.

Weaknesses and Risks

1. Financial opacity undermines credibility. SaferAI rates companies on transparency but doesn't practice it for its own finances. No published budget, no annual report, undisclosed funders. For an org whose value depends on being an independent, credible assessor, this is a significant vulnerability.

2. Advisory/rating dual role creates structural conflicts. The G42 case -- advising on a framework and then rating it -- mirrors the pre-2008 credit rating agency problem. As commercial revenue grows, this conflict could become systemic. SaferAI has no published COI policy to manage it.

3. Fabien Roger conflict is unaddressed structurally. Board member works at the highest-rated company. The behavioral evidence suggests independence (Anthropic was downgraded), but the structural conflict should be addressed through formal recusal procedures, not just good behavior.

4. Public-information-only methodology has fundamental limitations. SaferAI rates what companies CLAIM they do, not what they ACTUALLY do. The frontier auditing paper argues that meaningful assurance requires deep access to non-public information. SaferAI's approach maps to the lowest assurance level in that framework.

5. Funder concentration risk. Jaan Tallinn appears to be the largest funding source across multiple vehicles. Single-individual dependency is fragile.

6. European focus in an increasingly US/China-dominated field. SaferAI's deepest engagement is with EU institutions. The EU AI Act matters, but the most consequential AI development happens at US labs and increasingly Chinese ones. US regulatory rollback and Chinese opacity limit the reach of EU-centric standards.

7. No demonstrated causal impact. After two rating rounds, there's no evidence that SaferAI specifically caused companies to improve (vs. other pressures driving the same changes). Two companies' scores actually declined. Without causal impact evidence, the ratings are informative but not demonstrably effective.

Cross-References

• FLI (Future of Life Institute): Direct partner. SaferAI provides the Safety Frameworks domain analysis for the FLI AI Safety Index. FLI provides a larger platform that amplifies SaferAI's ratings.
• FAR.AI: CBRN consortium partner with FAR.AI leading. Collaborative relationship.
• METR: Collaborated on G42 framework. METR does model evaluations (different from SaferAI's framework-level assessment).
• GovAI: Malcolm Murray is a Research Affiliate. GovAI is a CBRN consortium partner.
• Anthropic: Fabien Roger dual role. SaferAI rates Anthropic; Anthropic generally scores highest.
• EffiSciences/CeSIA: French AI safety ecosystem that Simeon helped build. Relationship unclear.
• Open Philanthropy/CG: Notably absent. SaferAI has zero grants from the largest AI safety funder.
• Credit rating agencies (analogy): SaferAI's model closely mirrors S&P/Moody's in structure and faces the same structural risks -- particularly the consulting-and-rating dual role.

What Would Change This Assessment

• SaferAI publishing a comprehensive annual report with full financial disclosure would significantly increase confidence in independence. This is the single most impactful thing they could do.
• SaferAI publishing a formal COI policy addressing Fabien Roger, advisory relationships, and future conflicts would address the governance gap.
• Evidence that a company changed specific practices citing SaferAI ratings as the reason would demonstrate causal impact, not just correlation.
• A substantive public critique of SaferAI's methodology by a credible external party would test whether their framework holds up under scrutiny.
• Loss of the EU CBRN contract or failure to renew would suggest SaferAI's work isn't valued by the regulators it's trying to influence.
• SaferAI gaining access to non-public company information (moving up the AAL ladder in the frontier auditing framework) would substantially strengthen their assessment capability.
• A company threatening or taking legal action over a rating would test SaferAI's institutional resilience and independence.

Self-Critique

What sources should I have checked but didn't?

• French-language podcasts (The Flares, Le Futurologue) -- could contain additional candid material
• Le Monde coverage referenced in bios -- could not be found
• Manifund project page comments -- could not be fetched (429 errors)
• Euractiv article -- blocked by site
• LinkedIn discussions about SaferAI

Where is this analysis potentially biased?

• I may be overly critical of the governance gaps because I'm primed to look for conflicts of interest by the analyst instructions. The advisory/rating dual role is a genuine concern but SaferAI is very early-stage and may address it as they mature.
• I may be underweighting the value of SaferAI's standards work because it's less visible than the ratings. Standards participation is harder to evaluate from the outside but may be the highest-impact activity.
• The absence of criticism could be over-interpreted. Small orgs often don't attract critics; absence of evidence isn't evidence of problems.

What would a thoughtful person who disagrees say? "SaferAI is one of the only organizations actually trying to build the measurement infrastructure that AI regulation will need. You're criticizing them for governance issues that every 3-year-old nonprofit has, while ignoring that they've already achieved EU government contracts, FLI partnerships, standards body seats, and TIME coverage with 15 people. The Fabien Roger conflict is managed in practice even if not in policy. The financial opacity is a French nonprofit norm, not a red flag. The real question is whether they'll scale fast enough to matter, not whether their COI policy is published."

What's my single weakest claim? My claim that the advisory/rating dual role is structurally concerning analogous to pre-2008 credit rating agencies. The analogy may be too strong -- credit rating agencies were operating in a $600B+ CDO market with massive financial stakes; SaferAI is a tiny nonprofit whose ratings don't directly drive investment decisions. The incentive pressure is orders of magnitude smaller.

What information would most change my view? SaferAI's complete funder list and budget. If it turns out that SaferAI is primarily funded by entities with no financial interest in AI companies (e.g., exclusively Jaan Tallinn and similar EA funders), the independence concern weakens substantially. If it turns out that AI companies themselves are significant funders, the concern strengthens dramatically.