Aligned AI

Stated Theory of Change

Original (2022-2023): Concept extrapolation solves the core alignment problem. AIs that can correctly generalize human concepts and values across changing environments will remain aligned even as their capabilities grow. Build ACE as the practical implementation, demonstrate on benchmarks, then apply to real-world systems. Armstrong's mechanism: "If the goals change as the world model changes, then you are naturally solving part of the alignment problem."

Current (2025-2026): Near-term AI harms are the priority because AGI via LLMs is impossible and the chatbot industry is collapsing. Build Canvas as a consumer product that embodies alignment principles: non-chatbot AI, offline operation, human agency over algorithmic engagement, prevention of parasocial relationships. The company's alignment expertise informs product design rather than producing alignment research for the field.

Revealed Theory of Change

Actions tell a different story from either stated narrative:

1. The research output has shifted from fundamental to applied. The 2022-2023 papers (concept extrapolation, CoinRun, bias measurement) were advancing an alignment research agenda. The 2025-2026 papers (DATDP jailbreak defense, AI Chaperones, emergent misalignment analysis) are practical safety engineering. This isn't bad work -- DATDP is genuinely useful -- but it represents a retreat from the founding ambition of solving the "very center" of alignment.

2. The team composition has shifted toward product. Hiring a social anthropologist (Jocelyn Kelley) for market research signals consumer product priorities. No visible alignment researcher hires.

3. The blog has become a commentary outlet. Recent posts (Grok, LLM pain, chess) are analysis/opinion pieces about other companies' failures rather than advancing the company's own research program.

4. Canvas displaces alignment research as the organizational focus. The homepage now leads with Canvas. The Wikimedia partnership, the market research, the product design -- these are the activities consuming the team's limited resources.

The revealed theory of change is: Aligned AI is becoming a consumer tech company that applies safety-first design principles, informed by but no longer advancing alignment research.

Key Assumptions

Assumption 1: Concept extrapolation is central to alignment.

• Evidence for: Armstrong's theoretical arguments are sophisticated; the CoinRun result demonstrates the basic mechanism works; analogies to physics and moral philosophy are compelling.
• Evidence against: The alignment field has not engaged with this claim. RLHF, constitutional AI, and scalable oversight approaches have attracted far more research attention. Concept extrapolation may be important but not uniquely central.
• Testable? Partially -- you could test whether ACE-like algorithms help with increasingly complex alignment benchmarks. Not yet tested.
• If wrong: The founding theory of change is a dead end, and the pivot to near-term safety products was correct for the wrong reason.

Assumption 2: LLMs cannot achieve AGI and the chatbot industry will collapse.

• Evidence for: Gorman's arguments about scaling law exhaustion have partial support from 2024 reporting. Her probability-compounding argument against scaling inference has logical structure.
• Evidence against: Reasoning models (o1, o3, DeepSeek) showed significant capability improvements post-scaling-law concerns. "Glorified autocomplete" is reductive about emergent capabilities. "AI winter" predictions have been wrong many times before.
• Testable? Yes, within 2-3 years.
• If wrong: The Canvas pivot is based on a misdiagnosis, and the company has abandoned alignment research just as AI systems become more capable and harder to align.

Assumption 3: A hardware+software offline consumer ecosystem can compete commercially.

• Evidence for: Real parental anxiety about children's technology use. Wikimedia partnership provides credible content. Product philosophy is genuinely distinctive (offline-first, non-chatbot AI).
• Evidence against: Hardware+software consumer products require significant capital, supply chain, and distribution -- none of which a 2-10 person company has demonstrated. Competing with Apple, Google, and Amazon in family tech is extraordinarily difficult.
• Testable? Yes, by Canvas adoption within 12-18 months of launch.
• If wrong: The company burns through remaining capital without achieving product-market fit.

Strengths

1. Armstrong's intellectual depth is genuine. His pre-founding work (safely interruptible agents, corrigibility, low-impact AIs) is foundational to the field. The concept extrapolation theory, even if not central to alignment, contributes genuinely novel ideas about how values fragment and recombine. His 10% self-assessment of direct success probability is rare intellectual honesty.

2. The practical safety tools are real. DATDP (jailbreak defense), AI Chaperones (parasocial prevention), GPT-Eliezer (prompt evaluation), faAIr (bias measurement) -- these are concrete, open-sourced contributions that others can build on. The CoinRun result, while modest, is a legitimate benchmark improvement.

3. The Canvas product philosophy is distinctive. Non-chatbot AI, offline-first, friction-by-design, human agency over algorithmic engagement -- this is a genuinely different approach to consumer AI that could serve as a demonstration of alignment-informed product design.

4. Mission-lock commitment. As a benefit corporation with a charter-level mission clause, the company has at least stated its intent to avoid pure commercial drift. This is more structural commitment than most startups make.

5. Policy engagement is real. Gorman's EU/UN/OECD advisory work and the EU AI Act manipulation papers contribute to governance, not just research.

Weaknesses and Risks

1. The pivot looks like a retreat disguised as strategy. Moving from "solving the very center of alignment" to "offline family AI ecosystem" is a massive scope reduction. The narrative that "chatbots are dying so we're building something better" is convenient but untested. The most parsimonious explanation combines genuine belief-change with resource constraints: they couldn't raise alignment-research-scale funding, and near-term consumer products are more fundable.

2. The alignment research community has not engaged. Zero CG/Open Phil funding, no top-venue publications, limited citations, no engagement from major alignment labs. This is not definitive proof the work is wrong, but the revealed-preference signal from the ecosystem that funds and evaluates alignment research is strongly negative.

3. Financial viability is uncertain. ~$730K raised over 4 years with no visible revenue. Canvas is a hardware+software product requiring manufacturing, supply chain, and go-to-market -- all capital-intensive. The Jan 2025 share allotment suggests bridge funding. It's unclear how the company survives long enough to ship Canvas.

4. Gorman's contrarianism may be overdone. "LLMs are glorified autocomplete" was a defensible position in 2023 but looks increasingly strained given o1/o3/DeepSeek reasoning improvements. Dismissing the entire industry while building a product that presumably uses LLM components (Canvas's AI features) creates internal tension. A CEO who sees AI winter coming while raising money for an AI product faces a credibility challenge.

5. Concept extrapolation has not scaled. After 4 years, demonstrations remain at image classification and video game benchmark level. The gap between "ACE gets 72% on CoinRun" and "safely extrapolate human values for superintelligent AI" is vast, and no progress on closing it has been published since late 2023.

6. Team is extremely thin. 2-10 people attempting to simultaneously conduct alignment research, build a hardware consumer product, write papers, and advise governments. This is a recipe for doing many things poorly rather than one thing well.

Cross-References

• FAR.AI (Adam Gleave): Advisor relationship. Both work on alignment, but FAR.AI has maintained focus on alignment evaluations and has CG/Open Phil funding. Contrast in trajectories is instructive.
• MIRI: Armstrong's intellectual ancestry. His "Smarter Than Us" was a MIRI publication. But MIRI's approach (theoretical deconfusion) diverges from Aligned AI's practical/product approach.
• Anthropic/DeepMind alignment teams: Armstrong co-authored with Jan Leike (now Anthropic) and Laurent Orseau (DeepMind). These labs have vastly more resources to pursue alignment research. The question of whether a 5-person startup can contribute meaningfully alongside billion-dollar labs is fundamental.
• Calm technology movement: Canvas's design philosophy ("calm, intentional technology") echoes the calm technology principles of Amber Case. More of a product design reference than an alignment one.
• AI safety startup landscape: Companies like Redwood Research, ARC, and METR have maintained alignment research focus with CG/Open Phil funding. Aligned AI's pivot away from research toward consumer product is unusual in this cohort.

What Would Change This Assessment

• Canvas ships and gains traction. If Canvas actually reaches market and demonstrates that alignment-informed product design resonates with consumers, the pivot narrative becomes "correct strategic foresight" rather than "retreat."
• Concept extrapolation publishes at a top venue. A NeurIPS or ICML paper showing ACE-style methods working on harder benchmarks or real-world systems would validate the research program.
• CG/Open Phil funds them. Would signal that the alignment funding ecosystem sees value in the approach.
• A major alignment researcher engages with concept extrapolation. Even a critical engagement would indicate the idea is taken seriously.
• Gorman's AI winter prediction proves correct. If the LLM industry contracts significantly, her positioning looks prescient.
• Canvas fails and the company returns to alignment research. Would suggest the pivot was a survival strategy, not a genuine belief change.

Self-Critique

• I may be unfairly harsh on the pivot. My assessment pattern-matches "alignment startup pivots to consumer product" to "retreat," but there's a genuine case that near-term harms (parasocial relationships, cognitive deskilling in children) are important and neglected, and that Canvas addresses real needs.
• I lack the EA Forum launch post and AMA. These would likely contain the most detailed articulation of the original theory of change and the community's initial reception. Without them, I'm reconstructing the early narrative from secondary sources.
• I'm uncertain about financial health. UK company accounts provide almost no information. The company may have secured more funding than visible, or may have consulting/contract revenue not publicly disclosed.
• My assessment of Gorman's contrarian views is from within the AI-capability-is-real mainstream. If she's right that LLMs can't achieve AGI and an AI winter is coming, my entire framing is wrong.
• Weakest claim: My claim that "the alignment research community has not engaged" is based on absence of evidence (no citations, no funding, no public discussion) which could simply reflect the company being too small, not the work being low-quality. Armstrong's pre-founding work was excellent; there's no reason to assume his company work is inferior simply because it's less cited.
• What would most change my view: Access to the EA Forum launch post and AMA, or a conversation with Armstrong about whether concept extrapolation research continues privately. Evidence that Canvas has a realistic path to market would also significantly update my assessment.