Ought / Elicit

Stated Theory of Change

Ought/Elicit claims two mechanisms for reducing AI risk:

1. Improving epistemics: By making high-quality, evidence-based reasoning cheaper and faster, Elicit helps researchers (including alignment researchers), policymakers, and decision-makers navigate the transition to advanced AI more wisely.

2. Pioneering process supervision: By building a commercially successful product using process-based (decomposed, transparent) architectures, Elicit creates an "existence proof" that this approach works. Other companies copy it out of self-interest. The landscape of AI development shifts toward more alignable architectures.

The causal chain: Elicit demonstrates factored cognition works commercially -> Other companies adopt similar approaches -> The default AI architecture becomes more transparent and supervisable -> Advanced AI systems are built on process-based foundations -> Alignment is easier.

Revealed Theory of Change

What Ought/Elicit's actions suggest about what they actually believe and optimize for:

Actions consistent with stated theory:

• Genuine commitment to process-based architecture (task decomposition, citations, transparent reasoning) even when it's harder to build.
• Published Factored Verification paper that others can adopt.
• Open-sourced ICE framework.
• Andreas's "Will We Get Wise Enough Fast Enough?" talk shows genuine concern about whether epistemics can keep pace with capability.
• Modest compensation during nonprofit years ($120-167K for the CEO) suggests real mission motivation.

Actions that complicate the stated theory:

• The product is domain-agnostic -- it accelerates ALL research equally. There is no mechanism to differentially accelerate safety work.
• Primary growth has been in biomedicine, engineering, and humanities -- not AI safety research.
• The PBC transition was driven by practical necessity (FTX collapse threatened funding model, product needed VC to scale) as much as strategic vision.
• Publication output has shifted from theoretical safety work to product-oriented research. The factored cognition research program was effectively abandoned in favor of building Elicit.
• No evidence of efforts to attract AI safety researchers as a user segment.

The honest picture: Elicit is a genuine, well-built product that embodies process-based principles. The founders are sincere about the safety connection. But the causal chain from "Elicit succeeds" to "AI risk reduced" involves multiple untested assumptions, and the product development has naturally pulled the org away from direct safety work toward commercial growth.

Key Assumptions

Assumption 1: Process-based systems can remain competitive with end-to-end systems.

• Evidence for: Elicit's product success with 400K+ users and $18-22M ARR suggests the approach works for current tasks.
• Evidence against: The "bitter lesson" of ML is that end-to-end optimization eventually wins. Every previous structured approach in vision, NLP, and games was eventually superseded.
• Testable: Yes, on a 2-5 year horizon. If frontier models can do systematic reviews end-to-end better than decomposed approaches, the assumption fails.
• If wrong: Elicit either adapts (abandoning process-based approach) or loses market position. Either outcome undermines the safety theory of change.

Assumption 2: Commercial success of process-based systems shifts the broader AI landscape.

• Evidence for: Labs have expressed interest in process supervision. OpenAI published a process-based supervision paper.
• Evidence against: Labs' expressed interest has not translated into meaningful investment. End-to-end training remains overwhelmingly dominant. Elicit is tiny relative to the frontier lab ecosystem.
• Testable: Partially. We can observe whether labs actually adopt process-based approaches at scale.
• If wrong: Elicit is a nice research tool that has no impact on how AI systems are actually built.

Assumption 3: Faster, better reasoning tools are net positive for AI safety.

• Evidence for: Better decision-making is plausibly good in expectation. Systematic reviews of AI governance options, risk assessments, etc. could be valuable.
• Evidence against: Elicit accelerates ALL research, including capabilities research. If it makes it 20% faster to survey the literature on scaling laws, that helps capabilities researchers too. No differential mechanism exists.
• Testable: In principle, but very difficult -- would require measuring downstream effects on safety vs. capabilities research.
• If wrong: Elicit is a general productivity tool that modestly accelerates the AI race.

Assumption 4: The nonprofit-to-PBC transition preserved the safety mission.

• Evidence for: PBC structure, stated mission, founder credibility, clean financial structure (no lab dependencies).
• Evidence against: VC investors optimize for growth and returns. The PBC benefit obligation is not enforceable in the way nonprofit status is. Commercial pressure to add features, expand markets, and grow revenue will increasingly compete with safety-oriented architecture decisions.
• Testable: Yes, by observing whether Elicit maintains its commitment to transparency as competitive pressure increases.
• If wrong: Elicit becomes a standard AI SaaS product that no longer embodies process-based principles.

Strengths

1. Intellectual coherence: The factored cognition -> process supervision -> Elicit product arc is genuinely coherent. Unlike many orgs that retrofit safety narratives onto existing work, Ought's product is a natural expression of its founding research vision.

2. Clean financial structure: No lab funding, no compute dependency, no board overlap with frontier labs. This is rare among orgs claiming to do AI safety work while also building AI products.

3. Founder credibility: Andreas has deep technical credentials (MIT PhD, Stanford postdoc, 20+ publications) and has been working on this problem for over a decade. The below-market compensation during nonprofit years is consistent with genuine mission motivation.

4. Real product-market fit: 400K+ MAU, $18-22M ARR, $100M valuation, word-of-mouth growth. This is not vaporware -- the product genuinely works and users pay for it.

5. Concrete safety-relevant contribution: Factored Verification is a practical technique for reducing hallucinations that others can and do adopt. This is more concrete than most safety org outputs.

6. Strong board: Owain Evans (TruthfulQA, Emergent Misalignment) brings world-class AI safety credibility to the Ought board.

Weaknesses and Risks

1. The "conjunctive" problem: The theory of change requires multiple independent assumptions to all hold simultaneously. Process-based systems must be competitive AND commercially successful AND adopted by others AND this must shift the AI development landscape AND the net effect must be safety-positive. Each conjunction reduces probability.

2. No differential safety mechanism: Elicit accelerates all research equally. There is no reason to believe it disproportionately helps safety researchers. The most safety-positive version of Elicit would specifically prioritize safety applications, but this would conflict with commercial growth incentives.

3. The bitter lesson looms: End-to-end ML systems have historically replaced every structured approach. If GPT-5 or equivalent can do systematic reviews end-to-end better than Elicit's decomposed approach, the core product thesis and the safety thesis collapse simultaneously.

4. Scale mismatch: Elicit is a ~25-person company with $31M in VC funding. Frontier labs spend billions. The notion that Elicit's success will meaningfully shift how labs build AI systems is plausible but requires extraordinary influence relative to scale.

5. Mission drift risk: The transition from nonprofit safety research to VC-backed SaaS product is a classic path toward mission drift. The PBC structure provides some protection but not enforcement. As revenue grows, the pressure to optimize for growth over safety-oriented architecture will increase.

6. Owen Cotton-Barratt on the board: His 2023 EA controversy (resigned from all other EA positions) could become a reputational liability if Ought reactivates or the board becomes publicly visible.

Cross-References

• Paul Christiano / ARC: Christiano's Iterated Amplification framework is the intellectual ancestor of factored cognition. He was on Ought's board until May 2024. ARC's research on scalable oversight is theoretically complementary to Elicit's practical approach.
• Owain Evans / Truthful AI: Evans's research on truthfulness, emergent misalignment, and LLM behavior is directly relevant to Elicit's mission. He bridges the gap between Ought's board and frontier safety research.
• William Saunders -> OpenAI: Saunders moved from Ought to OpenAI's alignment team, suggesting Ought's research environment prepared people for serious safety work. His subsequent resignation over safety concerns at OpenAI is independently informative.
• Competitors: Consensus AI (consensus views across studies), Semantic Scholar (free, larger database), Scite (citation context). Elicit differentiates on depth of processing and transparent methodology. None of the competitors make safety claims.
• Open Philanthropy: Funded Ought's early years under "Navigating Transformative AI." The fact that Open Phil classified Ought as AI safety work (not general science) is noteworthy.

What Would Change This Assessment

Upward updates:

• Evidence that AI safety researchers are a significant and growing Elicit user segment.
• A frontier lab explicitly adopts process-based supervision citing Elicit's influence.
• Elicit introduces differential features for safety research (e.g., free/discounted access for alignment researchers).
• Independent accuracy audit confirms Elicit's claims and shows clear advantage over end-to-end approaches.

Downward updates:

• Elicit abandons process-based architecture for end-to-end approaches to stay competitive.
• Evidence that Elicit primarily accelerates capabilities research.
• VC investors push for growth that conflicts with safety-oriented design choices.
• Frontier models make the decomposed approach obsolete for Elicit's core use cases.
• The Ought nonprofit's equity stake is diluted or structured in ways that prevent it from funding future safety work.

Self-Critique

What's weakest in this analysis:

• I have limited access to the actual forum discussions about Ought's theory of change (forum domains are restricted). The "conjunctive theory" critique and the "speeding up science" concern are from web search summaries, not full reading.
• I may be over-crediting the alignment windfall argument because it's intellectually compelling even if empirically unvalidated.
• The competitive landscape analysis is shallow -- I have not used Elicit, Consensus, or Semantic Scholar to directly compare.

What a thoughtful disagreer would say: "You're treating Elicit's safety claims too seriously. It's a research tool SaaS company. The safety framing is a founder narrative that helped raise philanthropic funding early on, and it persists because it differentiates from competitors and appeals to EA-adjacent investors. The actual safety impact is approximately zero -- no frontier lab is going to change its architecture because Elicit has 400K users."

My single weakest claim: That Elicit's success will "shift the landscape" toward process-based AI development. This is the most speculative link in the theory of change, and I found no evidence that it's happening.

What would most change my view: Evidence (positive or negative) about whether AI safety researchers actually use Elicit, and whether Elicit's process-based approach is actually being adopted by other companies building AI systems.