Meaning Alignment Institute

Stated Theory of Change

MAI claims the fundamental obstacle to beneficial AI is conceptual: we don't understand what human values actually are, so we can't align AI to them. Their stated mechanism:

1. Define values rigorously (not preferences, not slogans, not rules -- constitutive attentional policies grounded in philosophical tradition)
2. Elicit values democratically at scale using LLM-powered interviews
3. Aggregate values into a moral graph that surfaces wisdom rather than majority opinion
4. Use this moral graph to train AI models, redesign economic mechanisms, and upgrade democratic institutions
5. This produces AI systems with "model integrity" -- coherent, inspectable values that guide behavior in novel situations

The theory rests on a chain of philosophical claims: that values are distinct from preferences, that they can be elicited through careful interviewing, that wisdom can be aggregated through a graph structure, and that this can be operationalized for AI training. Each link in the chain is necessary for the theory to hold.

Revealed Theory of Change

MAI's actions largely track their stated theory, with some notable divergences:

Consistent: The research output -- the values paper, the 500-person experiment, the moral graph implementation, the WiseLlama prototype -- all directly implement the stated theory. The open-source ethos (code + data publicly available) reinforces the stated goal of building public goods.

Divergent -- scope expansion: The trajectory from DFT (2023) to model integrity (2024) to full-stack alignment (2025) to market intermediaries (2025-2026) reveals a pattern of expanding ambition faster than execution capacity. Each new frame is intellectually coherent but none of the previous ones has been validated at scale before the next one launches.

Divergent -- network vs. execution: The Full-Stack Alignment paper with 30+ co-authors looks more like academic coalition-building than technical execution. The gap between "we have a 3-person team" and "we want to redesign markets, democracy, and AI alignment" is extreme. The revealed theory of change may be: "build intellectual consensus among academics, then let that consensus influence how institutions approach these problems" -- which is different from "build and deploy aligned AI systems."

Divergent -- prioritization: MAI appears to prioritize theoretical completeness over empirical validation. They published the full-stack alignment paper before validating the basic DFT approach against standard baselines. The absence of benchmarks is not an oversight -- it reflects a genuine belief that getting the conceptual framework right matters more than incremental empirical results.

Key Assumptions

Assumption 1: The "what to align to" question is as important as the "how to align" question.

• Evidence for: Anthropic's struggles with Claude's constitution, OpenAI's inability to specify values beyond vague helpfulness, the entire Constitutional AI debate
• Evidence against: If inner alignment / deceptive alignment is the binding constraint, better alignment targets don't help. If capabilities outpace alignment regardless of target quality, the work is moot.
• Testable: Yes -- compare outcomes of models trained on moral graphs vs. RLHF vs. CAI. MAI has not done this.
• If wrong: MAI's work becomes a philosophical contribution with no practical impact on AI safety.

Assumption 2: Values are distinct from preferences and can be reliably elicited.

• Evidence for: The 500-person experiment showed 97% of participants could articulate values in the required format, and most endorsed the result as representing "what they care about."
• Evidence against: Single study, Americans only, paid Prolific workers. The moral disagreement paper argues these kinds of crowdsourced approaches systematically fail on genuinely controversial questions. The GPT-4-generated stories could bias what gets classified as "wiser."
• Testable: Yes -- cross-cultural replications, adversarial testing with motivated reasoners, comparison with expert-driven approaches.
• If wrong: Moral graphs capture something, but not genuinely "deeper" values -- just a different transformation of preferences.

Assumption 3: Wisdom can be aggregated democratically (the moral graph concept).

• Evidence for: 89% fairness endorsement, expert values surfacing through PageRank.
• Evidence against: The Substack critic's point -- "Have massive public processes historically proven to result in wisdom?" Small sample, no adversarial participants, no real-world consequences. The graph had only one cycle in 3 scenarios.
• Testable: Yes -- larger experiments with real stakes, adversarial participants, more divisive scenarios.
• If wrong: The moral graph is a clever data structure but doesn't actually aggregate toward wisdom.

Assumption 4: The approach scales to institutional transformation.

• Evidence for: The theoretical framework is coherent across multiple domains.
• Evidence against: Zero adoption after 2.5 years. Klingefjord himself says they lack capacity to even run a homelessness policy pilot. 3-person team with $265K confirmed funding.
• Testable: Partially -- the market intermediaries pilot could provide early evidence.
• If wrong: MAI remains an intellectual contribution that never translates to practice.

Strengths

1. Genuinely novel intellectual contribution. The distinction between values and preferences, operationalized through constitutive attentional policies and the moral graph, is a real conceptual advance. It draws on serious philosophy (Taylor, Chang, Sen) that most alignment researchers ignore.

2. Ryan Lowe's involvement. A co-creator of RLHF leaving OpenAI to join a 3-person nonprofit is extremely strong evidence that someone who has seen the limitations of RLHF from the inside believes MAI's approach is worth pursuing.

3. No conflicts of interest. MAI has no financial ties to frontier labs, no compute dependencies, no board overlaps. This gives their critique of RLHF/CAI unusual credibility.

4. Promising empirical signals. The 500-person pilot produced surprisingly strong results: bipartisan convergence on contentious topics, high fairness endorsement, expertise surfacing through PageRank. These deserve follow-up.

5. Academic network. 30+ co-authors on FSA from MIT, Oxford, CMU, etc. demonstrates the intellectual framework is taken seriously by academics across disciplines.

Weaknesses and Risks

1. No empirical validation against baselines. This is the critical weakness. After 2.5 years, there is no formal comparison showing moral-graph-aligned models are better (by any metric) than RLHF or CAI-aligned models. The claim that DFT is "better" is theoretical.

2. Scope far exceeds capacity. A 3-person team with ~$265K in confirmed funding is attempting to: redefine human values, build democratic AI alignment processes, redesign economic mechanisms, and reform democratic institutions. Something has to give.

3. No adoption path. The OpenAI connection dissolved. No lab, company, or government has expressed interest in deploying the approach. The "if you build a good enough framework, they will come" strategy is high-risk.

4. The technical alignment community isn't paying attention. MAI's LW posts get modest engagement. No prominent alignment researcher has engaged substantively with the moral graph concept. This could mean the approach is seen as philosophically interesting but irrelevant to the safety-critical questions that dominate the field.

5. GPT-4 dependency in the elicitation process. The entire moral graph construction depends on GPT-4 generating value transition stories and mediating the chat process. This introduces model bias into what's supposed to be a democratic process. MAI acknowledges this limitation but has no solution.

6. Does not engage with catastrophic risk. MAI's theory of change assumes a world where AI integration is gradual enough for democratic processes to keep pace. If AI capabilities accelerate faster than institutions can adapt, the "full-stack" approach may be too slow.

Cross-References

Complementary to: Anthropic (who could use moral graphs instead of vague constitutional principles), Collective Intelligence Project (overlapping goals, different methods), RadicalxChange (overlapping institutional reform vision).

Tension with: MIRI/machine alignment approaches (MAI focuses on outer alignment targets; MIRI focuses on inner alignment/deception), ARC Evals/METR (evaluations focus on dangerous capabilities, not value alignment quality).

Gap-filling: MAI fills a genuine gap in the alignment ecosystem -- almost no one else is doing rigorous philosophical work on what alignment targets should look like. Most orgs take the alignment target as given (human preferences/feedback) and work on the technical challenge of getting AI to pursue it.

What Would Change This Assessment

Strongly positive updates:

• A formal evaluation showing moral-graph-aligned models outperform RLHF/CAI models on value alignment metrics, resistance to manipulation, or user trust
• Adoption by a frontier lab or major platform of any MAI technology
• Cross-cultural replication of the 500-person experiment with similar results
• Significant new funding (>$1M) enabling larger experiments and hiring

Strongly negative updates:

• Cross-cultural replication showing the approach fails outside WEIRD populations
• Evidence that moral graphs are trivially gameable by motivated actors
• Key personnel departures (especially Ryan Lowe)
• Continued absence of adoption after 5+ years of operation

Self-Critique

What sources should I have checked but didn't?

• The full Jim Rutt podcast transcript from Jan 2023 (pre-MAI founding) and Oct 2025 (FSA discussion) -- I read portions but not the full transcripts. These could contain more candid statements about MAI's challenges.
• The Clearer Thinking podcast with Joe Edelman -- not fully read.
• Whether any technical alignment researchers have discussed MAI privately (Twitter/X threads, private forums).

Where is this analysis potentially biased?

• I may be giving too much credit to the philosophical sophistication of the framework. Elegant philosophy doesn't guarantee practical relevance. The alignment community's silence could reflect genuine assessment that the work doesn't address binding constraints, not merely ignorance.
• I may be overweighting Ryan Lowe's involvement as a validation signal. People leave jobs for many reasons.

What would a thoughtful person who disagrees say? "MAI is doing philosophy of value, not AI safety. The binding constraint on alignment is not 'what to align to' but 'how to prevent deceptive optimization.' Even if MAI perfectly solves the alignment target problem, it doesn't help if we can't ensure AI systems actually pursue the target. Meanwhile, the scope expansion (markets! democracy! economy!) looks like a nonprofit that can't show impact on its core mission, so it keeps reframing the mission to be bigger."

Single weakest claim? That Ryan Lowe's departure from OpenAI validates MAI's approach. It's a strong signal but I may be overweighting it. Lowe may have left OpenAI for personal reasons (burnout, disagreement with leadership) and chosen MAI as a comfortable landing spot rather than a conviction bet.

What information would most change my view? A rigorous, pre-registered evaluation comparing models trained on moral graphs against RLHF/CAI models. If MAI produces better models by standard metrics AND novel metrics, my assessment would shift dramatically positive. If the comparison shows no meaningful difference, my assessment would shift sharply negative.