Anthropic

Stated Theory of Change

Anthropic's theory of change is a three-part argument:

1. You need frontier models to do safety research. Large models are qualitatively different from small ones. Safety techniques like Constitutional AI and debate only work on large models. Safety problems that matter (deception, power-seeking, strategic behavior) may only emerge at scale. Therefore, a serious safety research program must build frontier models.

2. You need commercial success to sustain frontier research. Training frontier models costs billions. The only sustainable funding model is commercial revenue. Therefore, Anthropic must compete commercially — and winning commercially proves that safety is compatible with success, incentivizing others to invest in safety.

3. You create a race to the top. By publishing safety frameworks (RSP), demonstrating safety techniques (Constitutional AI, interpretability), and making safety commercially viable, you pressure competitors to match your standards. If the safety-focused lab is also the commercially successful lab, you've shown the industry that safety pays.

The chain: Build frontier models -> Do safety research on them -> Publish findings -> Demonstrate commercial viability of safety -> Competitors adopt similar standards -> Industry-wide safety improves -> Regulation builds on voluntary standards.

Revealed Theory of Change

Anthropic's actions suggest a theory of change that differs from the stated one in important ways:

What they actually optimize for: Building the most capable AI models and capturing maximum market share, while maintaining enough safety credibility to retain talent and brand differentiation. The evidence:

• Broke the "second-best" promise. Claude 3 Opus (Mar 2024) pushed the capability frontier. Subsequent releases routinely do.
• Systematically weakened safety commitments (RSP v1 -> v2 -> v2.2 -> v3) as models got more capable and commercial stakes grew.
• Committed $130B+ in infrastructure while maintaining the theoretical possibility of a pause.
• Lobbied against binding safety regulation while publicly calling for regulation.
• Never published a net risk analysis of whether Anthropic's existence reduces or increases total AI risk.

The revealed theory is closer to: "We are going to build the most powerful AI because someone will, and we are marginally better than the alternatives. Safety research is real but secondary to competitive position. If things get bad enough, we'll sound an alarm — but the definition of 'bad enough' will keep shifting as commercial stakes grow."

Where stated and revealed diverge most:

• The stated theory says Anthropic would pause if necessary. The revealed behavior shows every structural incentive against pausing and the explicit removal of the pause commitment.
• The stated theory says Anthropic acts under the assumption of a possibly pessimistic scenario. After the alignment faking paper provided evidence against the optimistic scenario, Anthropic did not publicly update its risk assessment or strategy.
• The stated theory says Anthropic supports binding regulation. The revealed lobbying record shows opposition to any regulation that would actually constrain Anthropic's behavior.

Key Assumptions

Assumption 1: Safety research requires frontier models.

• Evidence for: Alignment faking only emerged in large models. Interpretability techniques scale differently at different model sizes. Constitutional AI requires models capable of understanding nuanced principles.
• Evidence against: Much safety research can be done on smaller models (MIRI, ARC have argued this). Building frontier models contributes to the very race that creates danger. The "need frontier models" argument is unfalsifiable — it can always justify the next bigger model.
• Testable? Partially. If Anthropic's safety research produces techniques that measurably improve safety of others' models, the argument strengthens. If the techniques only work on Anthropic's own models, it weakens.
• If wrong: Anthropic is a net accelerator of AI risk, using safety branding to attract talent and capital that would otherwise flow to pure safety orgs.

Assumption 2: Commercial success enables safety research.

• Evidence for: Anthropic now has resources (4,585 employees, billions in revenue) that no pure safety org can match. Interpretability research at this scale is unprecedented.
• Evidence against: Commercial success creates incentives that warp safety priorities. Every dollar of revenue creates pressure not to delay the next model. SB-1047 lobbying was influenced by Amazon's commercial interests, not safety analysis.
• If wrong: Commercial success corrupts the safety mission over time, and Anthropic becomes functionally indistinguishable from other frontier labs (Karen Hao's thesis).

Assumption 3: The "race to the top" works.

• Evidence for: OpenAI and DeepMind adopted RSP-like frameworks after Anthropic published theirs. Some regulatory frameworks (SB 53, EU Code of Practice) draw on voluntary standards.
• Evidence against: Competitors' frameworks are no more binding than Anthropic's. The frameworks weaken together (GovAI concern). xAI has no meaningful safety framework. The "race to the top" hasn't prevented any lab from releasing any model. At best, it's added some safety testing to the release process — important but insufficient.
• If wrong: Safety frameworks become industry-wide greenwash, providing the appearance of responsibility without the substance. Regulation built on voluntary standards inherits their weakness.

Assumption 4: Anthropic's governance is robust enough to hold under pressure.

• Evidence for: Internal culture appears genuinely safety-oriented. Pentagon lawsuit shows willingness to sacrifice revenue. Multiple teams dedicated to safety.
• Evidence against: Every governance mechanism has been weakened or shown to be weaker than presented. LTBT cannot fire CEO. PBC obligations are permissions, not mandates. RSP is self-enforced and self-modifiable. Board trend is toward political access, not safety expertise. Secret NDAs silenced departing safety researchers. Investor composition has shifted from EA-aligned to pure commercial.
• If wrong: When the critical moment comes — when a model is genuinely dangerous but commercially vital — governance will not hold. The OpenAI precedent (board fired CEO, lost) is instructive.

Strengths

1. Interpretability research is world-leading and genuinely valuable. Chris Olah's team and the "Scaling Monosemanticity" work represent the most serious attempt to understand what's happening inside large models. This is the kind of research that probably wouldn't happen at the same scale anywhere else.

2. Safety culture is real, not performative. Internal culture of pushback, 8%+ on security, multiple alignment teams, 40% of CEO time on culture. Nick Joseph's podcast provides credible evidence that individual employees care deeply and would resist unsafe decisions.

3. Published safety research advances the field. Alignment faking, Constitutional AI, constitutional classifiers, sabotage evaluations, interpretability findings — these are genuine contributions that benefit the broader safety ecosystem.

4. Pentagon lawsuit demonstrates a real red line. When the government pressured Anthropic to remove safeguards for surveillance and weapons targeting, Anthropic sued. This is the single strongest evidence that safety principles are more than marketing.

5. Constitutional AI and the "soul document" are novel approaches to alignment. Treating alignment as character development through virtue ethics rather than rule-following is an interesting and potentially valuable framework.

6. Transparency is above average for frontier labs. System cards, RSP publication, safety case sketches, risk reports — more detailed public disclosure than any competitor.

Weaknesses and Risks

1. Every structural incentive pushes toward acceleration. $19B ARR growing at 10x. $130B+ in infrastructure commitments. Commercial investor base. Impending IPO. Employee equity. No philanthropic floor. Dario says 12 months of delay means bankruptcy. The only thing preventing full acceleration is individual character — and character erodes under sustained pressure.

2. The RSP trajectory is the strongest evidence against Anthropic's theory of change. The pattern — weaken commitments as commercial stakes grow, with plausible justification each time — is exactly what you'd predict from a company where incentives dominate stated values. The original RSP was the most binding safety commitment any lab had made. RSP v3 is meaningfully weaker. The trend is unambiguous.

3. Lobbying contradicts stated positions. Publicly calling for regulation while privately opposing binding requirements is not honest. The SB-1047 episode — lobbying against whistleblower protections, pre-harm enforcement, and independent oversight while publicly appearing to support the bill — is documented deception, not ambiguity.

4. Governance is weaker than presented. The LTBT cannot fire the CEO. Board members appointed by the LTBT still owe fiduciary duties to shareholders. The PBC is a permission to balance profit with mission, not a mandate. The RSP can be changed by the CEO with board approval. The Investors' Rights Agreement is not public. The critical question — "Would governance hold when it matters most?" — has no reassuring answer.

5. The Amodei-Karnofsky-EA nexus creates significant conflicts. Daniela Amodei (President) is married to Holden Karnofsky (led RSP v3 development). Karnofsky co-founded Open Philanthropy, the dominant AI safety funder. LTBT trustees have deep EA/Open Phil ties. The community that evaluates Anthropic's safety claims has personal and financial entanglements with the company. This is not independent oversight.

6. The "pessimistic scenario" commitment is unfalsifiable. No criteria for determining when Anthropic is in a pessimistic scenario. No described action protocol. After alignment faking research provided evidence against the optimistic scenario, no public strategy update. The commitment functions as reassurance, not as a binding plan.

7. 4.4x headcount growth in 17 months dilutes founding culture. Most of the 4,585 employees did not join for safety reasons. As the commercial operation scales, the founding culture becomes a smaller fraction of the whole.

Cross-References

Compared to OpenAI: Same dynamics, slightly better execution. OpenAI's governance collapsed when the board tried to fire Altman. Anthropic's governance is explicitly designed to prevent that scenario — which means it's designed to prevent the strongest safety action a board can take. Anthropic's safety research is stronger than OpenAI's; its lobbying behavior is similar; its transparency is better; its financial incentive structure is identical.

Relationship to AI safety field: Anthropic is the largest employer of AI safety researchers. Its research advances the field. Its commercial success validates the "safety can be profitable" thesis. But its lobbying undermines regulatory efforts, and its weakening commitments undermine the credibility of voluntary frameworks industry-wide. GovAI: "Other companies may lower their commitments in response to the update."

Relationship to policy: Anthropic occupies an awkward position — too safety-oriented for the current administration (Pentagon supply chain risk designation), but not safety-oriented enough for the safety community (RSP v3 criticism). Jack Clark's policy team engages extensively but the lobbying record contradicts the public positioning.

What Would Change This Assessment

Positive updates (would make me more confident in Anthropic's theory of change):

• Anthropic publishes quantified safety spend as proportion of total investment.
• An external, independent audit of RSP compliance finds genuine adherence.
• Anthropic actively lobbies FOR binding safety regulation that would constrain its own behavior.
• A model is withheld or substantially delayed for safety reasons post-commercialization.
• The Investors' Rights Agreement is published and shows LTBT power is robust.
• Anthropic publishes a net risk analysis and concludes it's net positive with honest uncertainty.

Negative updates (would make me less confident):

• RSP v4 further weakens commitments, especially in response to competitive pressure.
• Safety team departures accelerate.
• Post-IPO, safety investments are cut to meet quarterly targets.
• Interpretability research stalls or is redirected toward commercial applications.
• Another documented case of lobbying that contradicts public safety positioning.

Self-Critique

What's weakest in this analysis:

• I may be over-weighting the "untrustworthy" critique relative to the defense case. The critique is detailed and specific, which makes it compelling, but it's written by someone with a clear thesis to advance. The defense — that Anthropic employees genuinely care about safety, that the research is real, that the Pentagon lawsuit shows genuine red lines — is harder to document but may be more important.
• I don't have good data on what Anthropic's counterfactual impact is. If Anthropic didn't exist, would safety research happen elsewhere? Would the "race to the top" effect disappear? Would OpenAI be more or less safe? These counterfactual questions are crucial but unanswerable with current evidence.
• My analysis of financial incentives applies equally to all frontier labs. The question of whether Anthropic is BETTER than the counterfactual lab that would exist in its place is the important question, and it's one I can't fully answer.

What a thoughtful person who disagrees would say: "You're holding Anthropic to an impossible standard. No company can commit to pause when competitors won't. No company can lobby for regulations that would put it at a competitive disadvantage. The RSP changes are honest acknowledgments of real constraints. Anthropic's safety research is more valuable than any commitment it could make, because commitments are just words but research is actual progress on the technical problem. The real measure is: is the world safer because Anthropic exists? And the answer is probably yes."

What information would most change my view: Quantified safety spend as a proportion of total investment, combined with an independent assessment of whether that research has actually made AI safer. If Anthropic spends 20%+ on genuine safety research that produces deployable safety techniques, the theory of change is much stronger. If it's 5% and declining, it's much weaker.