Cadenza Labs

Stated Theory of Change

Cadenza Labs exists to "research, benchmark and develop robust lie detectors for LLMs." The implicit causal chain is:

1. AI deception is a critical alignment risk -- if models can lie undetectably, all safety measures that rely on model reports become unreliable.
2. Current lie detection methods (probes, LLM judges, behavioral classifiers) have been tested only in narrow settings and fail when challenged with diverse lie types.
3. By creating comprehensive benchmarks that expose these failures, Cadenza forces the field to confront the actual difficulty of the problem.
4. By organizing competitions, Cadenza accelerates development of better methods through structured adversarial testing (red teams create lies, blue teams detect them).
5. Eventually, reliable lie detectors would enable deployment monitoring, evaluation safeguarding, and alignment auditing -- three concrete safety applications.

The deeper theoretical foundation (from Kaarel Hanmi's 2023 post) frames this as a special case of finding concepts in neural networks by their structural properties. Truth/honesty is the first and most safety-relevant concept to search for, with the long-term vision of being able to find any concept (utility, goals, beliefs about other agents) in model internals.

Revealed Theory of Change

What Cadenza actually spends its limited resources on reveals a more specific theory:

Benchmarking as field-building. The team's primary output is evaluation infrastructure -- Liars' Bench, CCS-Lib, the competition. They are not trying to build a working lie detector; they are trying to systematically demonstrate that current detectors don't work, and to create the tools that enable others to build better ones. This is fundamentally an infrastructure play.

Taxonomy over engineering. The two-axis categorization of lies (object of belief x reason for lying) is arguably more valuable than any specific detection method. By providing a shared vocabulary for discussing different kinds of AI dishonesty, Cadenza shapes how the field thinks about the problem.

Competition as leverage. The Schmidt Sciences / NDIF competition lets a 3-person team punch far above its weight. Rather than hiring more researchers, they're using competition structure to get dozens of external teams producing lie datasets and detection methods.

The gap between stated and revealed: The original vision (from Kaarel's framework) was about unsupervised concept discovery -- finding truth representations in models without supervision. The current work has shifted toward supervised benchmarking and evaluation. This is pragmatic but narrower. The team no longer appears to be pursuing the "Ramsification" approach to finding concepts by their shape.

Key Assumptions

Assumption 1: Models have recoverable internal representations of truth/belief that are distinct from output behavior.

• Evidence for: Burns et al. (2022) DLK results, various probing papers showing truth-correlated directions in activation space, representation engineering results.
• Evidence against: Levinstein & Herrmann argue probes learn spurious correlations. The "model might not represent 'is this true or not'" (Collin Burns' own caveat).
• Testable? Yes -- this is exactly what probing research tests. Cadenza's own results show probes barely beat chance on diverse lie types.
• If wrong: The entire probing-based approach fails. Behavioral methods (LLM-as-a-Judge) become the only viable approach, which is what Liars' Bench actually found -- the black-box judge outperformed white-box probes.

Assumption 2: Lie detection at the assertion level is sufficient for safety.

• Evidence for: Collin Burns' argument that "if the model is totally honest, we're in okay shape" is compelling for the simplest threat models.
• Evidence against: Berger (2026) shows models can deceive without lying. A model that tells only technically-true-but-misleading statements evades assertion-based detectors entirely.
• Testable? Yes -- Berger's paper already demonstrates this failure mode.
• If wrong: Cadenza needs to broaden from "lie detection" to "deception detection," which requires fundamentally different methods (probing for intent, second-order beliefs, or communicative function rather than truth/falsity).

Assumption 3: Open-weight model lie detection results transfer to frontier closed models.

• Evidence for: Anthropic's paper uses similar taxonomies and finds similar patterns. Some lie types are hard to detect regardless of model size.
• Evidence against: Cadenza only works with open-weight models (Llama, Qwen, Mistral, Gemma). Frontier model behavior may be qualitatively different. Labs doing their own internal lie detection have advantages Cadenza cannot access.
• If wrong: Cadenza's benchmarks become exercises for the open-weight community but don't address the hardest safety problems at the frontier.

Assumption 4: Competition structure advances the field effectively.

• Evidence for: ML competitions (ImageNet, SQuAD, etc.) have historically driven rapid progress. The red team / blue team structure is well-suited to adversarial evaluation.
• Evidence against: Deception detection may be fundamentally different from standard ML tasks. The "Building Better Deception Probes" finding that 70.6% of variance comes from prompt choice suggests that "general-purpose detectors" may be impossible, which undermines the competition framing.
• If wrong: The competition produces interesting datasets but no generalizable detection methods.

Strengths

1. Fills a genuine gap. Before Liars' Bench, lie detection methods were validated on narrow, unrealistic datasets. Cadenza created the first comprehensive benchmark with diverse on-policy lie types.

2. Intellectual clarity. The two-axis taxonomy of lies is clean, operationalized, and adopted by others (including Anthropic). Good conceptual work enables faster progress by the entire field.

3. Punching above weight. A 3-person team with under $200K in confirmed funding produced an EMNLP paper, a PNAS paper, a major benchmark, and organized a competition backed by Schmidt Sciences. The output-to-resource ratio is exceptional.

4. Strategic positioning. By building evaluation infrastructure rather than competing on specific methods, Cadenza occupies a non-rival niche. When better detectors are built, they will be validated on Cadenza's benchmarks.

5. Right collaborator network. Samuel Marks (Anthropic), Alex Mallen (Redwood), Erik Jenner (DeepMind), Chris Cundy (FAR.AI) provide connections across the safety ecosystem. The Liars' Bench acknowledgments include many top interpretability researchers.

6. Low incentive risk. No commercial products, no lab funding dependencies, no equity investors. Cadenza's incentives are aligned with its mission in a way that larger, more complex organizations often are not.

Weaknesses and Risks

1. Fragile organization. A 3-person team with no legal entity, no formal governance, undisclosed finances, and 75% founding team turnover is structurally fragile. If Walter Laurito left or became unavailable, it is unclear whether Cadenza would continue to exist.

2. The "deception without lying" blind spot. Berger (2026) demonstrated that the assertion-based definition misses a critical class of AI deception. Cadenza has not publicly engaged with this critique. Their competition still uses the assertion-based definition. If the field pivots to broader deception detection, Cadenza's framing becomes outdated.

3. Resource asymmetry with labs. Anthropic's Samuel Marks team published a paper doing everything Cadenza does plus more, using frontier models and proprietary fine-tuning. Labs can always one-up independent benchmarking work. Cadenza's value depends on being an independent evaluation authority -- but they may be squeezed out if labs internalize this function.

4. Chronic underfunding. The team has operated for 2+ years on under $200K in confirmed external funding. Team members subsidize participation through PhD positions. This is not sustainable for the kind of systematic benchmarking work they aspire to do.

5. Missing theory-of-impact. There is no clear path from "competition produces better detectors" to "detectors are deployed in production AI systems." The competition has no formal relationship with any lab's safety team or deployment process.

6. No candid public articulation. Zero podcast appearances, zero blog posts explaining their strategy, zero public reflections on setbacks or pivots. The analysis must rely entirely on inferring intentions from papers and a 3-page website.

Cross-References

Complementary to Anthropic's safety team: Cadenza provides independent benchmarking that Anthropic's internal team cannot credibly provide (independence matters for evaluation). Samuel Marks' involvement bridges both.

Similar niche to METR (formerly ARC Evals): Both provide evaluation infrastructure. METR focuses on dangerous capability evaluations; Cadenza focuses on deception detection evaluations. The approaches are complementary.

Overlapping with Apollo Research: Apollo works on detecting deception in LLMs but approaches it through different methods (strategic deception in agentic settings, in-context scheming). Apollo is better-funded and larger.

Related to EleutherAI's ELK project: Cadenza's Walter Laurito contributes to EleutherAI's CCS-Lib. Both work on the same foundational tools for probing model knowledge.

Potential beneficiary of Schmidt Sciences Interpretability RFP: The $300K-$1M RFP (due May 2026) specifically seeks deception detection research. Cadenza is well-positioned to apply. This could be a transformative funding event for the org.

What Would Change This Assessment

• Upward: Cadenza secures substantial funding (>$300K) from Schmidt Sciences or another funder, enabling full-time researchers and addressing the sustainability concern. The competition produces a detection method that meaningfully outperforms existing baselines across diverse lie types. Cadenza addresses the deception-without-lying critique by expanding their framework.

• Downward: Walter Laurito leaves or reduces involvement. The competition fails to attract strong teams or produces only incremental results. Another group (Anthropic, Apollo, a well-funded academic lab) subsumes the benchmarking function. The field consensus shifts to "probing-based lie detection is a dead end."

• Neutral but important: Cadenza formalizes as a legal entity and publishes finances. Team members give interviews explaining their strategy. Kaarel Hanmi returns to active involvement, reinvigorating the theoretical side.

Self-Critique

What sources should I have checked but didn't?

• Manifund page and donor list (blocked by 429 errors)
• Kieron Kretschmar's thesis (would reveal depth of his distributional-shift work)
• Any private communications or grant applications that might explain team transitions
• Conference presentation videos if any exist

Where is this analysis potentially biased?

• I may be overly sympathetic to a small underfunded team doing careful work. The scrappiness can obscure genuine strategic problems.
• I may be projecting a theory of change onto a team that has not articulated one. They may simply be researchers doing interesting work without a strategic plan for impact.
• The "deception without lying" critique may be less fatal than I present it, if Cadenza sees lie detection as one component rather than a complete solution. But without interviews or strategy documents, I cannot know.

What would a thoughtful person who disagrees say? "Cadenza is an informal group of part-time PhD students who happened to produce one good benchmark paper. The competition is organized by Schmidt Sciences and NDIF -- Cadenza is along for the ride. They have no path to deployment, no legal entity, no money, and no evidence they can solve the hard parts of the problem. The theoretical vision from Kaarel is gone and what remains is competent but not field-shaping benchmarking work."

What's my single weakest claim? That Cadenza's benchmarking work will actually improve AI safety outcomes. It is entirely possible that Liars' Bench becomes a useful academic exercise without ever influencing how labs build or deploy models. The missing link between "benchmark exists" and "safety improves" is the weakest part of the entire theory of change.

What information would most change my view? A candid interview with Walter Laurito explaining: (1) what happened with Kaarel, Kay, and Georgios, (2) where the money actually comes from, (3) what the plan is for deploying competition results, and (4) whether they plan to address deception-without-lying. This single source would resolve most of the uncertainty in this analysis.