Cooperative AI Foundation

Stated Theory of Change

CAIF's stated theory of change is that AI safety requires not just alignment (individual agents doing what their operators intend) but also cooperative intelligence (agents solving multi-agent coordination problems). As AI systems move from passive tools to autonomous agents interacting with each other -- trading, negotiating, competing -- the interaction dynamics become a distinct source of catastrophic risk. Flash crashes, algorithmic collusion, arms race escalation, and institutional erosion can all occur even among perfectly aligned agents.

The mechanism: by funding foundational research on cooperative AI now (game theory for AI agents, benchmarks, evaluation methods), building a talent pipeline (PhD fellowships, summer schools, curriculum), and shaping governance frameworks (policy engagement on EU AI Act, US AI Action Plan), CAIF aims to ensure cooperative capabilities exist ahead of or alongside dangerous multi-agent deployment.

Founder Allan Dafoe frames this as a bet of similar magnitude to alignment: if we solve alignment but fail at global coordination, great powers deploy aligned AI against each other with catastrophic consequences. The bet is that dedicated investment in cooperative AI research -- analogous to inventing the seatbelt before the car -- is both important and neglected.

Revealed Theory of Change

CAIF's actions broadly match its stated mission, with some revealing patterns:

Research is purely academic. All 15 grants go to universities. No grants to policy organizations, advocacy groups, or AI labs. No tools or frameworks deployed in production. The revealed priority is building the intellectual foundations of cooperative AI as an academic field, not solving immediate practical problems.

Field-building dominates grant-making. CAIF's most successful programs are the PhD Fellowship (30 fellows across two cohorts), summer schools (65+ attendees/year), contests (1000+ participants total), and the online curriculum. These are classic field-building activities. By contrast, the grant program under-spent by 50% in 2024. CAIF is more effective at growing the number of researchers than at funding specific research projects.

The DeepMind connection is operational, not just structural. Beyond board seats (2/5 trustees), CAIF collaborates directly with DeepMind on the Concordia Contest, multiple DeepMind researchers co-author CAIF's flagship report, and Dafoe references DeepMind's work extensively when discussing cooperative AI. The revealed relationship is one of close partnership -- CAIF functions partly as an academic arm of DeepMind's multi-agent safety interests.

Policy work is secondary but growing. The TechPolicy Press article on EU AI Act gaps represents a shift from pure research to practical governance advocacy. This is recent (January 2026) and may signal CAIF expanding beyond its academic comfort zone.

The funder connection runs deep. Macroscopic Ventures' focus on s-risk, AI welfare, and suffering reduction shapes what CAIF cares about. The CLR research agenda (cooperation failures as a source of vast suffering) is CAIF's intellectual ancestor. Jesse Clifton bridging CLR/Macroscopic and CAIF is not incidental -- it's a deliberate alignment of research direction with funder philosophy.

Key Assumptions

1. Multi-agent AI deployment will be a major source of catastrophic risk, distinct from single-agent alignment failure.

• Evidence for: Flash crashes are real. Algorithmic collusion exists in pricing. Military AI brinksmanship is a credible concern. The EU AI Act Article 73 analysis shows regulatory gaps.
• Evidence against: Most current AI safety concerns center on single-agent misalignment, deception, and loss of control. Multi-agent deployment at dangerous scale remains hypothetical.
• Testable? Yes. As agentic AI deploys (2025-2027), we will see whether multi-agent incidents become a significant category. The EU AI Act reporting framework provides a measurement opportunity.
• If wrong: CAIF's work is not wasted (game theory and evaluation methods have broad utility) but its urgency claims are overblown.

2. Cooperative intelligence will not emerge automatically from scaling general AI capabilities.

• Evidence for: Current LLMs show cooperative behavior in simple games but are "not very strategic or goal-directed" (CAIF's own assessment). The relationship between general intelligence and cooperative intelligence is empirically unknown.
• Evidence against: Christiano and Shulman argue capabilities improvement will produce cooperative skills as a byproduct. General-purpose models are already better at many cooperation-relevant tasks than dedicated narrow models.
• Testable? Partially. Track whether frontier models' cooperative capabilities improve with scale without dedicated cooperative AI training.
• If wrong: CAIF becomes a nice-to-have accelerator rather than a necessary institution. The "super-cooperative AGI hypothesis" makes CAIF's mission less urgent.

3. Investing in cooperative capabilities is net positive for safety, not net negative.

• Evidence for: Historically, improved cooperation technology (trade, contracts, institutions) has been net positive for humanity. Dafoe argues that on net, "increasing everyone's cooperative capability" benefits prosocial actors more than antisocial ones.
• Evidence against: CAIF's own M-FOS research shows that opponent-shaping algorithms naturally learn to extort. Cooperative skills are dual-use: they enable collusion, exclusion, and manipulation as effectively as they enable cooperation. The MIRI concern is that cooperatively skilled AI before aligned AI gives AI systems tools to cooperate against humans.
• Testable? Difficult. This is fundamentally a bet about the ratio of prosocial to antisocial applications.
• If wrong: CAIF's research advances the capabilities of AI systems to manipulate multi-agent environments, which could be used against human interests.

4. Academic field-building is the right intervention at this stage of AI development.

• Evidence for: The cooperative AI research community is genuinely small. CAIF's programs (30 PhD fellows, 65+ summer school attendees, 1000+ contest participants) are building the researcher base that will be needed when multi-agent risks materialize.
• Evidence against: Agentic AI is deploying now. "Foundational not operational" (AIGL) may mean the field is not building tools fast enough to matter during the critical deployment window.
• Testable? Yes. Within 2-3 years, either CAIF's research translates into practical deployment guidance or it remains academic.
• If wrong: CAIF should shift toward applied research, tooling, and direct lab engagement.

Strengths

Intellectual coherence. CAIF has a clear, well-articulated theory of change that is distinct from mainstream alignment work. Dafoe's argument connecting technological determinism to cooperative AI is original and internally consistent.

Institutional honesty. The 2025 grants update is one of the most candid institutional self-assessments I have encountered across dozens of AI safety organizations. CAIF openly admits failures, diagnoses root causes, and proposes specific fixes. This level of self-criticism suggests good epistemic culture.

Talent pipeline. The PhD fellowship, summer schools, contests, curriculum, and now the Cape Town research fellowship constitute a genuine talent pipeline. If multi-agent safety becomes important, CAIF will have trained many of the researchers working on it.

DeepMind access. While this creates governance concerns, it also means CAIF has a direct line to one of the world's leading AI labs. Dafoe's dual role means cooperative AI research has a voice in DeepMind's strategic decisions.

Dual-use awareness. CAIF openly acknowledges that cooperative capabilities are dual-use and that their work could facilitate exploitation. The M-FOS research and the "Cooperative AI bet" framing (labeled a "hypothesis" rather than a certainty) show genuine engagement with the strongest objections.

Weaknesses and Risks

Single-funder dependency. 100% of CAIF's funding comes from Macroscopic Ventures. If Macroscopic changes priorities, CAIF has no fallback. This also means CAIF's research direction is shaped by one entity's philosophy (s-risk, suffering reduction, EA-aligned values). No funding diversification has been attempted or achieved in 4+ years.

DeepMind governance capture. 40% of the board (Dafoe + Graepel) works at Google DeepMind. The sole funder's representative (Clifton) also sits on the board as "Grantmaking Officer." There is no evidence of published conflict-of-interest procedures for these specific relationships. CAIF has not published anything critical of DeepMind. The concentration of board influence from one lab and one funder undermines independence claims.

Foundational not operational. CAIF's most significant outputs are taxonomies, frameworks, and papers. The AIGL review's critique -- "limited practical tooling or metrics" -- is directly relevant. The field-building strategy makes sense in theory, but agentic AI is deploying now. If CAIF's work does not translate into tools practitioners can use within the next 2-3 years, the critical deployment window may close.

Under-spending problem. Funding less than 50% of budgeted grants capacity in 2024 suggests either the field is too small to absorb the funding, the scope is too narrow, or the process is too burdensome. CAIF's response (narrower definitions, two-stage process) may help but may also further constrain the applicant pool.

Low community visibility. Only 2 forum posts across LW/EAF/AF. No Wikipedia article. Minimal public debate about cooperative AI as a field. This suggests CAIF has not penetrated the broader AI safety community's attention.

Absence of external criticism. A healthy research organization should attract substantive critics. CAIF's immunity to criticism could mean it is too small to matter, or it could mean the cooperative AI frame is positioned to avoid criticism (who argues against cooperation?). Either way, the absence of external challenge risks intellectual stagnation.

Cross-References

Complementary to alignment organizations (Anthropic, ARC, MIRI, Redwood): CAIF explicitly positions cooperative AI as complementary to alignment, not competing with it. The argument is that even if alignment is solved, multi-agent dynamics create distinct risks. If this framing is correct, CAIF fills a genuine gap.

Overlap with GovAI: Dafoe founded both organizations. GovAI focuses on AI governance broadly; CAIF focuses on multi-agent dynamics specifically. The research agendas overlap in policy areas (EU AI Act, international governance) but diverge technically (GovAI does more institutional analysis, CAIF does more game theory).

Connected to CLR and the s-risk community: CAIF's funding and intellectual lineage trace to the Center on Long-Term Risk and the broader suffering-reduction community. This positions CAIF at the intersection of AI safety and animal welfare/digital sentience concerns, which is unusual among AI safety organizations.

Google DeepMind connection: CAIF functions partly as an academic complement to DeepMind's multi-agent safety work. The Concordia Contest, shared personnel, and co-authored papers make this a functional partnership. This is distinct from arm's-length funder-grantee relationships.

FOCAL/CMU as a research hub: CMU's Foundations of Cooperative AI Lab (Vincent Conitzer) receives funding from both CAIF ($500K) and Macroscopic Ventures ($3M separately), making it the most heavily funded node in the cooperative AI research network.

What Would Change This Assessment

• Evidence that multi-agent incidents are a significant category of AI harm (flash crashes at scale, algorithmic collusion causing measurable economic damage, military AI brinksmanship) would strongly validate CAIF's theory of change.
• Evidence that cooperative capability scales automatically with general intelligence (frontier models becoming much better at multi-agent coordination without dedicated training) would undermine CAIF's neglectedness claim.
• CAIF producing practical tools adopted by AI labs (multi-agent risk evaluation frameworks, deployment guidelines, benchmarks used in production) would address the "foundational not operational" critique.
• CAIF publishing work that challenges DeepMind on any multi-agent risk would demonstrate governance independence.
• Funding diversification (any grant from CG/Open Phil, SFF, EA Funds, or government sources) would reduce single-funder risk.
• External criticism -- any substantive published challenge to CAIF's theory of change, effectiveness, or governance -- would either strengthen the organization through response or reveal weaknesses.

Self-Critique

What sources should I have checked but did not?

• The full 21-page 2024 Strategy PDF (unreadable) and the full Impact Report PDF likely contain more substantive self-assessment than the blog summaries.
• The Nature 2021 article by Dafoe et al. (paywalled) is the most prestigious articulation of the founding vision.
• Individual interviews or statements from Clifton about his dual role as funder-connected trustee and grantmaking officer.
• Any private communications between CAIF and Macroscopic Ventures about the endowment terms.

Where is this analysis potentially biased?

• I may over-weight the DeepMind governance concern because it fits a familiar "lab capture" narrative. CAIF's DeepMind connections could be purely beneficial -- providing access, legitimacy, and research collaboration without compromising independence.
• I may under-weight the importance of CAIF's field-building because it is hard to measure. The PhD fellows and summer school alumni may become the researchers who solve multi-agent safety problems in 5-10 years, which would make CAIF retrospectively one of the most important AI safety organizations.

What would a thoughtful person who disagrees say?

• "You're treating the DeepMind connection as a bug when it's the feature. CAIF exists to bridge academic research and frontier lab deployment. Having board members from the most important multi-agent AI lab in the world is exactly right."
• "You're treating 'foundational not operational' as a criticism when it's describing the correct phase. You don't build tools before you understand the problem. CAIF is doing theory now because that's what the field needs now."

What's my single weakest claim?

• That the absence of external criticism is a sign of insufficient community engagement rather than simply reflecting that CAIF is young, small, and non-controversial. A 6-person charity that has existed for 4 years and spends GBP 1.7M/year may simply be below the threshold for public criticism.

What information would most change my view?

• Evidence that CAIF's research has directly influenced any major AI lab's deployment decisions would shift my assessment from "promising but unproven" to "demonstrably impactful."