OpenMined

Stated Theory of Change

OpenMined's stated theory of change operates at two levels:

Near-term (funded by CG/OP): Build privacy-preserving infrastructure that enables external evaluation of frontier AI models. If evaluators can access models without seeing proprietary weights, and model developers can be evaluated without losing IP, then governments and researchers can hold AI developers accountable. This creates an accountability layer that makes safety commitments verifiable.

Long-term (Trask's full vision): Attribution-based control transforms AI from a "tool of central intelligence" into a "communication technology." When every AI prediction carries attribution to its data sources, users can verify and control what informs their AI outputs. This solves hallucinations, copyright, value alignment, concentration of power, and loss of control simultaneously, while unlocking "6 orders of magnitude more data" by incentivizing data owners to participate.

The causal chain: Build open-source PETs infrastructure -> Enable external evaluation of frontier models -> Create accountability for AI developers -> Make safety commitments verifiable -> Reduce catastrophic risk. In the long term: Shift AI architecture from parametric to retrieval-based -> Enable attribution-based control -> Decentralize AI power -> Make AI safe by design.

Revealed Theory of Change

Actions suggest a more nuanced picture:

What they actually do: OpenMined has successfully deployed PySyft for social media algorithm auditing (DailyMotion, LinkedIn, Reddit), cross-border statistical collaboration (UN), and one proof-of-concept secure enclave pilot with simulated data. The highest-impact deployments are in data governance, not AI safety.

Where the money goes: 5 paid staff on $623K total compensation, running deployments across multiple countries and building complex cryptographic infrastructure. The rest presumably goes to compute, subcontractors, and infrastructure costs associated with the grants. The volunteer model subsidizes what would otherwise require a much larger paid team.

What the trajectory reveals: The 2017 blockchain origins, the 2020 federated learning focus, the 2022 social media auditing work, and the 2024 secure enclave pivot form a pattern: OpenMined builds the best version of privacy-preserving computation it can with available technology, then repositions to match the highest-value application. The pivot to AI safety evaluation coincided precisely with the availability of CG/OP funding under "Navigating Transformative AI."

The gap: No frontier model has been evaluated through OpenMined infrastructure with real data. The Dec 2024 pilot used GPT-2 (a 2019 model) and a public dataset. The NIST CRADA is a research agreement, not an operational deployment. The distance between where OpenMined is and where it needs to be for the safety theory of change to work is large.

Key Assumptions

1. Evaluations reduce catastrophic risk.

• Evidence for: OP's entire evaluation portfolio assumes this. If-then commitments (Anthropic's RSP, etc.) require evaluations as triggers.
• Evidence against: Mukobi identifies 10 failure modes. Evaluations may not improve understanding (6 ways) or mitigation (4 ways). Current benchmarks saturate quickly. Labs can "teach to the test."
• Testable: Yes -- track whether evaluation results actually change lab behavior.
• If wrong: OpenMined's entire AI safety value proposition collapses. The infrastructure becomes useful for data governance and regulatory compliance but not for preventing catastrophic risk.

2. Secure enclaves provide sufficient security guarantees.

• Evidence for: Hardware-based encryption, attestation protocols, near-native performance. Used by Apple, deployed in Azure.
• Evidence against: TEE.fail defeats all three chipmakers' protections with 3-minute physical access. Side-channel attacks remain an active research area. Current TEEs limited to single physical servers.
• Testable: Yes -- ongoing security research will reveal whether TEEs can be hardened against physical attacks.
• If wrong: The multi-party governance model loses its trust anchor. If the enclave can be compromised, there's no guarantee that confidential data stays confidential.

3. Labs will voluntarily participate (or be compelled to).

• Evidence for: Anthropic participated in the UK AISI pilot. NIST signed a CRADA. EU AI Act and other regulation may mandate external evaluation.
• Evidence against: CSET paper notes "many model owners may not be so willing." No frontier lab has submitted real models for evaluation through OpenMined infrastructure. Current participation is for proofs of concept, not operational evaluations.
• Testable: Yes -- watch whether any lab actually uses this for real model evaluation.
• If wrong: The infrastructure exists but goes unused. Without regulatory mandates, the default is voluntary cooperation, which historically produces limited accountability.

4. Attribution-based control is technically achievable.

• Evidence for: Trask's thesis describes how existing techniques could be combined. Retrieval-based models (RETRO) demonstrate some aspects. Mixture of experts is a step toward partition.
• Evidence against: The thesis is a pre-print, not a demonstrated system. Scaling retrieval-based architectures to frontier capability is an open research problem. The claim that ABC solves value alignment conflates information attribution with goal specification.
• Testable: Partially -- retrieval-based model performance is measurable.
• If wrong: The grand vision fails but the near-term evaluation infrastructure remains useful.

5. Data owners will participate in OpenMined's network.

• Evidence for: Reddit, DailyMotion, LinkedIn, UN agencies, NIST have all participated. The value proposition for data owners is clear.
• Evidence against: These are pilot programs and research collaborations, not production deployments. No evidence of self-sustaining network effects.
• Testable: Yes -- measure growth in deployments and data owners.
• If wrong: The "internet for non-public information" vision doesn't materialize.

Strengths

Genuine technical capability. PySyft is real, deployed software with 9.8K GitHub stars. The CCIAO deployment demonstrated privacy-preserving auditing at real organizations. The CSET paper independently validates the approach. This is not vaporware.

Institutional partnerships. NIST CRADA, UK AISI pilot, UN PET Lab, Christchurch Call. These represent genuine engagement with governance institutions that could mandate evaluation.

Correct identification of a real gap. The inability to evaluate AI models without seeing proprietary information is a genuine barrier to accountability. OpenMined is the leading effort to close this gap.

Trask's intellectual range. The Interconnects interview reveals a thinker who connects hardware, software, policy, economics, and history. His analysis of GPU demand distortion and retrieval-based models shows genuine insight.

Open source commitment. Everything is Apache 2.0. This eliminates vendor lock-in concerns and enables community scrutiny of the codebase.

Weaknesses and Risks

The DeepMind conflict is disqualifying until addressed. The Executive Director works at a frontier lab whose models the org's infrastructure evaluates. This is not an edge case -- it is a fundamental governance failure for an organization claiming to enable independent evaluation. No conflict disclosure, no recusal policy, no public acknowledgment of the tension.

Zero engagement with the AI safety community. No LW/EA Forum presence, no 80K Hours appearance, no alignment research citations. The theory of change has not been vetted by the community that is most expert in evaluating AI risk reduction strategies. This is a $17M+ investment that has received zero community scrutiny.

Proof-of-concept to production gap. The secure enclave pilot used a 2019 model and public data. Frontier models require multi-node clusters that current TEEs cannot support. The TEE.fail attack undermines the core security guarantee. The gap between demonstration and operational safety infrastructure is large and growing as models scale.

Evaluation infrastructure does not equal safety. Even if perfectly implemented, secure enclave-based evaluation only works if: evaluations themselves are effective (contested by Mukobi et al.), labs voluntarily participate or are compelled, evaluation results actually change lab behavior, and the right things are being evaluated. OpenMined addresses the access problem but not the methodology, interpretation, or enforcement problems.

Financial fragility. 75% funding concentration in CG/OP, 100% grant-dependent, no earned revenue. Five paid staff. If CG priorities shift (as they have for other grantees), OpenMined's operations could collapse.

Governance immaturity. 501(c)(3) status since 2024. No public board list. No 990 filings. No Charity Navigator rating. A podcast host is the only publicly identified board member. Governance structures appear to be catching up with the pace of funding.

Cross-References

METR / Apollo Research / AISI: These organizations are the primary users of the evaluation infrastructure OpenMined builds. They focus on what to evaluate; OpenMined focuses on how to evaluate securely. Complementary if evaluation methodology matures.

Anthropic: Participated in the secure enclave pilot. If Anthropic scales this into real pre-release testing through OpenMined infrastructure, it validates the approach. But Trask's DeepMind employment creates a tension with evaluating Google's competitor.

Open Philanthropy / Coefficient Giving: The dominant funder. OP's own RFP on evaluation identifies three challenges; OpenMined addresses one (access). The other two (benchmarks, evaluation science) represent OP's recognition that infrastructure alone is insufficient.

Alignment research orgs (ARC, MIRI, Redwood): No connection. OpenMined operates at the governance/infrastructure layer, not the alignment theory layer. This is a valid specialization but means their theory of change depends entirely on evaluation-based approaches.

What Would Change This Assessment

Upward: A frontier lab submits a real model for evaluation through OpenMined infrastructure and the results inform a consequential safety decision. TEE.fail is mitigated by next-generation hardware. Trask publicly addresses the DeepMind conflict with a formal recusal policy. Regulatory mandates for external evaluation create guaranteed demand. Board composition includes independent governance expertise.

Downward: TEE vulnerabilities prove fundamental rather than fixable. No frontier lab adopts the infrastructure beyond pilots. Mukobi-style critiques gain empirical support showing evaluations don't change lab behavior. CG/OP funding doesn't renew. The ABC framework fails to produce working retrieval-based models with frontier capability.

Neutral but informative: First 990 filing reveals spending patterns. Board composition becomes public. OpenMined engages with LW/EA Forum and the theory of change faces community scrutiny.

Self-Critique

Weakest claim: My assessment that the ABC framework "conflates tractable problems with intractable ones" may be unfair. It's possible that attribution is a deeper unifying principle than I'm giving it credit for. I have not engaged deeply enough with the technical details of how retrieval-based architectures could provide formal safety guarantees to fully evaluate this claim.

Potential bias: I may be systematically undervaluing infrastructure work because it's less intellectually exciting than alignment theory. The history of technology suggests that infrastructure often matters more than anyone expects. The people who built HTTPS didn't need to understand cryptography theory -- they needed to ship reliable software. OpenMined might be in this category.

What I missed: I could not access Trask's Cohere talk (YouTube), the UK Parliament written evidence, or the Medium interview. These might contain important additional perspectives. I also have limited visibility into the actual security properties of the implementation, since no independent audit is public.

What a thoughtful disagreer would say: "You're applying AI safety community standards to an organization that deliberately operates outside that community. OpenMined's theory of change is about governance infrastructure, not alignment research. It should be evaluated against governance organizations (NIST, OECD, ISO) not against alignment orgs (MIRI, ARC). By that standard, a $17M org building infrastructure that NIST has formally adopted is an extraordinary success."

What would most change my view: Evidence that a frontier lab submitted a real model for evaluation through OpenMined infrastructure and the results actually changed a deployment decision. This would close the proof-of-concept to production gap and demonstrate the full causal chain.