Goodfire

Stated Theory of Change

Goodfire's stated theory of change has evolved significantly in its short existence and is now centered on intentional design: using interpretability tools to shape what AI models learn during training, not just inspect them afterward.

The causal chain: (1) Develop interpretability tools that extract meaningful semantic features from model internals --> (2) Use those features to decompose what a model is learning from each training example --> (3) Intervene on the training process to selectively promote or suppress specific learned behaviors --> (4) This produces models that are more aligned, less prone to hallucination, and more controllable --> (5) Simultaneously, use interpretability as a "microscope" to extract scientific knowledge from superhuman models.

Tom McGrath frames this as moving from selective breeding to genetic engineering for AI. The key principle is "don't fight backprop" -- rather than trying to override gradient descent, reshape the loss landscape so the model naturally learns what you want.

The safety-specific theory of change is: if we can understand and steer what models learn, we can build alignment into the training process rather than bolting it on afterward. This is complemented by the "train/test split" proposal -- use one set of interpretability techniques during training, a different set for auditing.

Revealed Theory of Change

Goodfire's actions reveal a more commercially pragmatic trajectory than the safety-first framing suggests:

Research evolution follows commercial viability. The company started with pure interpretability (SAEs, feature extraction) -- a safety tool. It evolved toward intentional design (training-time intervention) -- which is both a safety tool AND a capabilities accelerator. And it's simultaneously pursuing scientific abundance (knowledge extraction from models) -- which is primarily a commercial application with the strongest revenue path. The research trajectory moves steadily toward what generates enterprise value.

Customer portfolio reveals priorities. Rakuten (PII detection), Mayo Clinic (genomic medicine), Arc Institute (genomics), Prima Mente (Alzheimer's). These are commercial partnerships, not safety deployments. The Alzheimer's biomarker discovery is scientifically exciting but has no direct connection to reducing AI catastrophic risk.

The pivot from public API to partner-deployed platform concentrates access among paying enterprise customers and away from the broader safety research community. Open-source SAE releases partially offset this.

Funding trajectory drives toward commercial value. Seed-to-unicorn in 18 months with commercial VC investors (B Capital, Salesforce Ventures, Eric Schmidt) who expect returns. The board composition is opaque, but commercial investors dominate.

The revealed theory of change is: build a commercially successful interpretability platform that generates revenue from enterprise customers, while conducting foundational safety-relevant research that maintains credibility with the safety community and attracts top researchers. Safety research is a recruitment tool, a brand differentiator, and a genuine passion of the founders -- but it is not the revenue driver.

Key Assumptions

1. Interpretability will scale to frontier models.

• Evidence for: Progress from toy models to 70B and 671B parameter models. SAEs, SPD, circuit tracing all scaling up. Kimi K2 (trillion-param) activation harvesting demonstrated.
• Evidence against: No evidence that current techniques capture the computationally important aspects of frontier reasoning. The "Mind the Coherence Gap" evaluation found prompting outperforms feature steering. The 2023 critique notes: "Properties of models which are dangerous are not low-level features, but high-level behavioral abilities."
• Testable: Yes -- can Goodfire's tools detect or prevent a known failure mode in a frontier model that evals miss?
• If wrong: Goodfire becomes a niche scientific tool vendor, not a safety infrastructure company.

2. Intentional design won't undermine interpretability as a "test set."

• Evidence for: RLFR result shows probes remain valid after training on a frozen copy. The frozen-copy approach structurally avoids direct gradient flow through the probe.
• Evidence against: This is one result on one model on one task. OpenAI's obfuscated reward hacking paper shows models can learn to hide intent. JBlack's critique: "gradient-decomposition approach only works if interpretability is complete and accurate -- which it isn't."
• Testable: Yes -- do probes remain valid after many rounds of intentional design, on diverse tasks, at frontier scale?
• If wrong: Goodfire's core innovation is net-negative for safety, actively degrading the interpretability tools that others rely on for auditing.

3. A VC-funded PBC can sustain safety research under commercial pressure.

• Evidence for: Ho's genuine concern about AI risk (left a successful company). McGrath's research career dedicated to this. PBC structure. Anthropic endorsement.
• Evidence against: $1.25B valuation requires revenue growth. Investor base is entirely commercial. Board composition is opaque. PBC governance is legally weak. No ring-fenced safety research budget. Historical precedent (OpenAI's transition from nonprofit to capped-profit) shows mission drift.
• Testable: Yes -- does the ratio of safety research to commercial work change over time?
• If wrong: Goodfire becomes a commercial AI tooling company that maintains a safety veneer.

4. Third-party interpretability companies add value beyond in-house lab teams.

• Evidence for: Cross-model perspective, independence from any single lab's incentives, ability to serve multiple customers. McGrath argues for the value of independent research.
• Evidence against: Anthropic runs the world's largest interpretability team. Google DeepMind has significant interp efforts. Labs have access to their own models' full training data, compute, and architecture details that Goodfire doesn't.
• Testable: Does Goodfire's research output match or exceed what Anthropic's team produces? Do customers choose Goodfire over building in-house?
• If wrong: Goodfire's tools become redundant as major labs build superior in-house alternatives.

Strengths

World-class research team. Tom McGrath (founded DeepMind interp team), Lee Sharkey (SAE pioneer, Apollo co-founder), Nick Cammarata (OpenAI interp co-founder). This is arguably the strongest independent interpretability team outside the major labs.

Intellectually honest leadership. McGrath's intentional design manifesto is remarkably candid about risks, limitations, and open problems. He explicitly says the techniques "probably shouldn't be used on frontier models today." This is not the language of a company overselling its product.

Concrete empirical results. The RLFR hallucination reduction (58%, with probe transfer) is a real result, not just a vision document. The Alzheimer's biomarker discovery is genuine science. These are not trivially dismissable.

Unique positioning. Only commercial company with a serious argument for how interpretability translates to training-time alignment improvement. The intentional design vision is genuinely novel and potentially transformative if it works.

Speed of execution. Seed to unicorn in 18 months. From founding to published research results on hallucination reduction and scientific discovery within 20 months. This is exceptionally fast for a research-first company.

Weaknesses and Risks

The "Most Forbidden Technique" problem is real and underaddressed. The frozen-copy probe approach is a clever technical response, but it's one data point on one model. The fundamental concern -- that optimizing against interpretability tools degrades those tools -- has not been resolved. The train/test split proposal is "currently hypothetical." If intentional design goes wrong, it could actively harm the broader safety ecosystem by degrading interpretability's value as an auditing tool.

Commercial pressure will intensify. $1.25B valuation with commercial VCs demands revenue growth. The three pillars have very different commercial profiles: Scientific Abundance (pharma clients) generates revenue, Intentional Design (model developers) generates revenue, Understanding (pure research) does not. Expect the balance to shift toward commercial applications.

Governance is dangerously opaque. No public board composition. No detailed safety policy for intentional design. No external safety advisory board. No stated criteria for when these techniques are safe for frontier models. PBC structure is legally weak. This is inadequate for a company developing training-time AI intervention capabilities.

Single independent evaluation found prompting outperforms. The "Mind the Coherence Gap" evaluation is limited but directionally concerning. If feature steering doesn't reliably outperform prompting for behavior control, a large part of the product value proposition is in question.

Key claims lack independent validation. The RLFR hallucination result has no external replication. The intentional design algorithms are not fully published. The safety community cannot independently verify whether these techniques are net-positive or net-negative.

Potential for capabilities acceleration. Interpretability tools can be used for capabilities (optimizing models, reducing compute costs, improving training efficiency) as easily as for safety. Goodfire's tools in the hands of customers could accelerate capabilities without commensurate safety benefit.

Cross-References

Anthropic: Both investor and most relevant competitor. Anthropic's in-house team (Chris Olah, circuit tracing, attribution graphs) is the benchmark for interpretability research. Goodfire's independence is its value proposition -- cross-model, cross-architecture perspective. But if Anthropic's tools are superior for their own models, Goodfire's market narrows to non-Anthropic models.

Apollo Research: Lee Sharkey's former org. Apollo does safety evaluations (behavioral evals, scheming detection). Goodfire does interpretability-based understanding and intervention. Complementary rather than competing. Sharkey's move from Apollo to Goodfire suggests he sees more leverage in interpretability tools than in behavioral evals.

Timaeus: Focused on understanding learning dynamics ("developmental interpretability"). Goodfire cites their patterning work as inspiration. Complementary research direction. Timaeus is academic/nonprofit; Goodfire is commercial.

MIRI, ARC (Paul Christiano): Focused on theoretical alignment. Goodfire is more empirically driven. Different bets on what approach will work.

Transluce/Tilde/Wisent: Smaller competitors in commercial interpretability. Tilde ($8M from Khosla) and Wisent are early-stage. Goodfire has significant lead in funding, team, and research output.

What Would Change This Assessment

Upward:

• Independent replication of RLFR results at frontier scale with diverse tasks
• Publication of detailed safety policy with clear criteria for frontier deployment
• Transparent board composition with safety-focused members
• Evidence that intentional design techniques catch failure modes that behavioral evals miss
• Revenue growth that doesn't come at the expense of safety research investment

Downward:

• Safety researchers depart citing commercial pressure
• Intentional design techniques shown to degrade probe accuracy across models (not just frozen copy)
• Board composition reveals no safety-focused members
• Research output shifts predominantly toward commercial applications
• Major lab (Anthropic, DeepMind) builds superior in-house tools, reducing Goodfire to niche player

Self-Critique

Potential bias toward skepticism: I may be overweighting the structural criticisms (VC incentives, governance opacity) relative to the genuine technical accomplishments. The RLFR result is real. The Alzheimer's discovery is real. The team quality is extraordinary. A purely structural critique risks missing that this might simply be the most effective way to advance interpretability research given current institutional constraints.

Missing information that would change my view: Revenue data. Board composition. A detailed safety policy for intentional design. Any of these could significantly shift the assessment. The opaque governance could hide strong safeguards -- or weak ones. I don't know, and the absence of information is itself a finding.

Weakest claim: That the VC funding model will inevitably subordinate safety to commercial pressure. There are counterexamples (Anthropic itself is VC-funded with safety commitments). The PBC structure, while legally weak, does provide some protection. And the founders' genuine commitment to safety is evident in the evidence. My structural concern may be overweighted relative to the founders' actual intentions.

What a thoughtful disagreer would say: "You're applying nonprofit/EA standards to a for-profit company. Goodfire's approach -- making safety commercially viable -- is exactly how you scale safety research beyond the narrow EA funding base. The alternative (waiting for philanthropy to fund all safety work) doesn't scale. VC funding attracts talent that philanthropy can't. And the commercial customers are actually deploying interpretability in real systems, not just publishing papers."

Sources I should have checked but couldn't: DeepMind's SAE deprioritization article (failed to fetch). Haize Labs red-teaming blog (failed to fetch). Full text of OpenAI's obfuscated reward hacking paper. Any private communication between Anthropic and Goodfire about the investment terms.