FAR.AI

Stated Theory of Change

FAR.AI's stated theory has three layers:

1. The problem: All ML systems are adversarially exploitable. This exploitability breaks most alignment proposals because they depend on helper systems (reward models, oversight models, interpretability tools) that can themselves be hacked by the main system.

2. The solution: Develop safety techniques (defense-in-depth, robustness improvements, fault-tolerant alignment, lie detectors, red-teaming methodologies) that provide demonstrable guarantees.

3. The pipeline: Research alone is insufficient. FAR.AI spans the entire value chain from exploratory research through scaled engineering proof-of-concept to field building and policy advocacy. The explicit goal is: develop the technique, prove it works, get labs to adopt it, then get governments to mandate it.

The mechanism is engineering rigor applied to safety. Gleave's core claim is that current defense-in-depth failures stem from sloppy implementation rather than fundamental impossibility, and that with "proper planning, meticulous experimental design, and at least some willingness to accept performance trade-offs," the defense-in-depth plan can work.

Revealed Theory of Change

Actions largely align with stated theory, with some notable divergences:

Alignment: The research portfolio matches the stated agenda well. Adversarial robustness (Go attack, scaling laws), red-teaming (STACK, jailbreak-tuning), deception detection (lie detectors), and defense-in-depth evaluation all directly serve the stated theory.

Divergence 1 — Fault-tolerant alignment deprioritized. The 2023 vulnerability blog named fault-tolerant alignment as the area they were "most excited by." But actual research output has tilted heavily toward applied red-teaming and defense testing. This may reflect a pragmatic judgment that red-teaming has more immediate impact, but it means the most theoretically novel part of FAR.AI's original agenda has received less attention.

Divergence 2 — Field building has grown faster than research. CG grants include $6.65M for field building, $2.42M for communications, $1.7M for office space, and $12M for regranting — totaling $22.8M for non-research activities versus $28.7M for research (plus $2.16M general support and $1.3M for specific research projects). Almost half of CG funding goes to activities other than in-house research. FAR.AI is becoming as much a field-building and coordination organization as a research lab.

Divergence 3 — Government work is growing. The EU AI Act CBRN consortium is a significant commitment. Government contracts create different incentive structures than philanthropic grants — accountability to bureaucratic timelines rather than research novelty. This may be a strategic evolution toward becoming a "private sector quasi-regulatory body," as Labenz suggested.

Key Assumptions

1. Defense-in-depth can be made to work with careful engineering.

• Evidence for: STACK paper shows that implementation failures, not fundamental impossibility, are the main vulnerability. Lie detector research shows training conditions can make models genuinely more honest.
• Evidence against: Zvi Mowshowitz's critique that defenses will fail simultaneously. FAR.AI's own finding that under bad training conditions, lie detector training produces 95% deception rates. Adversarial robustness has been an open problem for a decade with limited progress.
• Testable: Yes. Can FAR.AI produce a defense that withstands a well-resourced red team?
• What changes if wrong: The entire research program loses its foundation. If defense-in-depth fundamentally cannot work, FAR.AI's full-stack approach has no end product to push through the pipeline.

2. There is enough time for iterative safety engineering.

• Evidence for: Gleave's median of 14 years for fully autonomous AI organizations.
• Evidence against: Many in the field estimate shorter timelines. GPT-5/Claude 4 already showing scheming behavior. If org-level AI arrives in 5 years instead of 14, the "iterate and improve" strategy is too slow.
• Testable: Partially — progress on agentic AI benchmarks provides ongoing data.
• What changes if wrong: The urgent priority shifts from iterative engineering to crude but immediate safety measures (pauses, compute governance, hard capability limits).

3. Labs will adopt safety techniques when shown they work.

• Evidence for: FAR.AI already collaborates with UK AISI and EU AI Office. Labs participate in defense-in-depth testing.
• Evidence against: Gleave himself notes "it really has been kind of just-in-time safety" and "there's a frog-boiling effect." Labs have economic incentives to cut corners. No evidence that FAR.AI's specific techniques have been adopted by major labs.
• Testable: Track whether labs implement FAR.AI's recommendations from the STACK paper.
• What changes if wrong: The "push through the pipeline" strategy fails at the adoption stage, and policy mandates become the only path.

4. A single organization can effectively span research, field building, consulting, regranting, and policy.

• Evidence for: FAR.AI's rapid growth and institutional endorsement (EU contract, diverse funders).
• Evidence against: No comparable organization has successfully operated at this breadth at scale. Each activity has different optimization pressures (research: novelty; consulting: client satisfaction; policy: consensus-building; regranting: evaluation rigor).
• Testable: Whether research quality is maintained as the org doubles in size.
• What changes if wrong: Spreading too thin produces adequate work in many areas but frontier work in none.

Strengths

Intellectual honesty. Gleave's willingness to commission and publish the Big Picture AI Safety study — which surfaced extensive criticism of the safety community — is unusual. His statement that "the AI safety community has quite possibly been harmful for the world" shows genuine epistemic humility. The org publishes research that undermines its own products (STACK breaking their own defense-in-depth pipeline).

Strong empirical foundations. The Go attack on KataGo is a landmark result: the first demonstration that superhuman AI systems have catastrophic adversarial vulnerabilities. The lie detector work provides actionable guidance. The research is grounded in experiments rather than theoretical speculation.

Strategic positioning. FAR.AI sits at the intersection of research, policy, and field building with relationships across governments (EU, UK AISI, Singapore CSA), academia (Berkeley, Cambridge), and labs (via consulting). The EU AI Act contract is a genuine institutional achievement for a 3-year-old organization.

Financial cushion. $22.5M in net assets provides substantial runway. Funding is diversifying beyond CG. Compensation is reasonable. The 10% consulting cap protects against lab capture.

Pragmatic worldview. Gleave's moderate skepticism of traditional AI risk arguments makes him credible to a broader audience than more alarmed safety researchers. The emphasis on engineering rigor over theoretical arguments appeals to ML researchers who may dismiss more abstract concerns.

Weaknesses and Risks

Extreme funder concentration. CG/OP accounts for $59.35M of total funding. While diversification is beginning, any significant shift in CG priorities would be existentially threatening to FAR.AI. CG is not just the primary funder — it funds research, regranting, events, office space, and communications. This level of dependence on a single philanthropic actor is a structural vulnerability.

Regranting opacity. A $12M discretionary fund with no public disclosure of recipients is a serious transparency gap. For an organization that has a detailed consulting transparency policy, the silence on regranting is conspicuous.

Breadth vs. depth tension. FAR.AI is simultaneously a research lab, an events organization, a coworking space, a regranting body, a policy shop, a consulting firm, and a communications team. With 15-30 researchers, doing all of this at frontier quality is extremely difficult. The risk is producing competent-but-not-groundbreaking work across many areas rather than field-defining work in any single one.

Governance concerns. Gleave's simultaneous board seats at FAR.AI, METR, SAIF, and LISA create concentrated influence across multiple safety organizations. The board composition beyond officers is unclear. For an org receiving $60M, governance transparency should be stronger.

Defense-in-depth may be a dead end. If Zvi Mowshowitz's critique is correct — that defenses fail correlated, not independently — then FAR.AI's core research agenda addresses a local optimum rather than the actual problem. The org's own STACK paper shows 71% bypass rates, which could be read as "defense-in-depth doesn't work even when carefully implemented."

Optimism bias. Gleave's relatively optimistic timelines (14-year median to org-level AI) and relatively optimistic assessment of defense-in-depth may create a false sense of security. If timelines are shorter and defenses weaker than assumed, the engineering-iteration approach fails.

Cross-References

METR: Gleave sits on METR's board. METR evaluates frontier models; FAR.AI red-teams and consults for frontier labs. This creates a web of influence across evaluation/testing organizations that should be watched.

Redwood Research: Co-organized ControlConf. FAR.AI actively builds the "AI control" field that Redwood pioneered. Complementary rather than competing.

UK AISI / EU AI Office: Collaborative relationships via STACK paper and CBRN consortium. FAR.AI is positioning itself as a trusted technical partner to government AI safety institutes.

Anthropic: Ethan Perez (Anthropic researcher) channels OP-funded research through FAR.AI. AI Safety Fund (partially funded by Anthropic) provides grants to FAR.AI. Defense-in-depth research tests Claude models. Multiple touchpoints.

CHAI (Berkeley): Gleave's PhD lineage under Stuart Russell. Several board members from Berkeley. Deep institutional roots.

Coefficient Giving: Primary funder by an extreme margin. CG's willingness to fund the full value chain through FAR.AI suggests very high confidence in Gleave and the team.

What Would Change This Assessment

• FAR.AI produces a defense that withstands a serious red team. If defense-in-depth can be shown to work against a well-resourced adversary (not just their own synthetic tests), this validates the core thesis. Currently the best result is an attack succeeding 71% of the time, which is not confidence-inspiring.

• A major lab publicly adopts a FAR.AI technique. The full-stack pipeline theory requires labs to actually use the research. Evidence of adoption would strongly validate the approach.

• Regranting recipients are disclosed. Transparency about the $12M program would address one of the biggest governance concerns.

• An independent evaluation finds the breadth is working. External evidence that quality is maintained across all activities would address the "spread too thin" concern.

• Gleave's timeline estimates prove wrong. If autonomous agents arrive in 2-3 years rather than 5-7, the iterative engineering approach may be insufficient.

• A notable researcher departs with public criticism. Would provide signal about internal culture and strategic disagreements currently invisible from outside.

Self-Critique

What I might be missing:

• The absence of forum discussion in the evidence means I may be missing community sentiment about FAR.AI — both praise and criticism.
• I may be under-weighting the significance of the EU AI Act contract. Government institutional endorsement of this kind is rare for a 3-year-old safety org and may signal quality that isn't visible in the available evidence.
• I may be over-weighting the "spread too thin" concern. Startup-style organizations do sometimes succeed at breadth through strong execution. The 30+ papers in 3 years suggests research output has not suffered yet.

Where this analysis is potentially biased:

• I am inclined to be skeptical of defense-in-depth because the evidence I read (STACK paper, Zvi's critique) highlights failures. But the same paper also contains Gleave's detailed response about when and why it can work under constrained threat models.
• The extreme funder concentration reads as alarming in isolation, but many successful research orgs (including early Anthropic, early OpenAI) were heavily dependent on single funders before diversifying. FAR.AI is actively diversifying.

Single weakest claim: That FAR.AI is "spread too thin." I have no direct evidence of quality suffering. The concern is based on the number of activities relative to team size, which is a reasonable inference but not an observed fact.

What would most change my view: Published regranting data showing thoughtful, well-evaluated grants would significantly increase my confidence in the organization's governance. Conversely, evidence that research quality has declined during the rapid growth phase would decrease it.