Haize Labs

Stated Theory of Change

Haize's stated theory of change is:

1. Frontier AI labs have misaligned incentives -- speed to market trumps safety testing.
2. Internal red-teaming is insufficient because labs can't objectively evaluate their own models.
3. An independent, automated third-party red-teaming company can fill this gap.
4. By discovering vulnerabilities before deployment, Haize reduces the harm from AI systems in the real world.
5. The mechanism: scalable automated adversarial testing reveals failure modes that labs miss or choose not to prioritize.

The causal chain from Haize's work to AI risk reduction runs: Haize discovers vulnerabilities --> labs receive vulnerability reports --> labs fix vulnerabilities --> models are safer when deployed --> real-world harm is reduced.

Revealed Theory of Change

The actions reveal a different, commercially pragmatic theory:

Red-teaming as market entry, reliability as the destination. The tagline evolution ("Bad day to be a language model" to "Deploy 99.9% Reliable AI"), the product suite expansion (from red-teaming to Judges, Monitor, Robustify, Sphynx), and the enterprise client base (Deloitte, MongoDB alongside Anthropic, OpenAI) all point in the same direction. Safety testing is the entry wedge; the real business is becoming the enterprise AI reliability/QA platform.

Research as marketing. The bijection learning paper (ICLR), the HuggingFace leaderboard, and the Constitutional Classifiers collaboration serve dual purposes: they advance genuine scientific knowledge AND they demonstrate technical credibility to potential clients and investors. The choice to publish offensive research openly (rather than keeping it proprietary) suggests the reputational value exceeds the competitive risk.

Independence is qualified. Despite Tang's claims of being "totally divorced from conflict of interest," the company is paid by the labs it tests, bound by NDAs, and admits it doesn't know whether its findings influence deployment decisions. The independence is commercial (they're a vendor), not structural (they're not an independent regulator or auditor).

Near-term focus. All published work addresses adversarial robustness, jailbreaking, and evaluation accuracy. No work on alignment, interpretability beyond attacks, governance frameworks, or long-term safety challenges. The research agenda is tightly focused on problems that have immediate commercial applications.

The revealed theory is not necessarily worse than the stated theory -- it may be more honest. A commercially sustainable company that incrementally improves AI safety testing is plausibly better than a safety organization that runs out of funding. But it means Haize's contribution to AI risk reduction is bounded by what's commercially viable, not by what's most impactful.

Key Assumptions

Assumption 1: Automated red-teaming finds vulnerabilities that matter.

• Evidence for: ACG, bijection learning, Cascade, and the o1/CC collaborations all demonstrate real vulnerabilities. The inverse scaling finding is a genuine contribution.
• Evidence against: The "security theater" paper argues red-teaming has never blocked a model release. Haize admits they don't know how findings are used. The most dangerous vulnerabilities (CBRN uplift) are tested by domain experts, not by Haize's automated methods.
• Testable: Yes -- track whether Haize-discovered vulnerabilities are fixed in subsequent model releases.
• If wrong: Haize provides expensive reassurance rather than genuine risk reduction.

Assumption 2: Third-party commercial testing is sufficiently independent.

• Evidence for: Haize has no equity in the labs it tests and no formal obligation to produce favorable results.
• Evidence against: Haize is paid by the labs, bound by NDAs, doesn't know how findings are used, has no whistleblower mechanism, and its angel investors include founders of potential customer companies.
• Testable: Would Haize publicly criticize a major client? Would they refuse to test a model they believed was too dangerous to release?
• If wrong: Haize is a paid vendor that provides a veneer of independent oversight, not genuine accountability.

Assumption 3: The enterprise reliability market will sustain a safety-oriented company.

• Evidence for: Regulatory tailwinds (EU AI Act requires adversarial testing), growing enterprise AI adoption, Haize's early multi-million dollar contracts.
• Evidence against: Labs may build superior in-house testing. Competitors (Robust Intelligence, Lakera, Patronus AI) crowd the market. Enterprise reliability may pull Haize away from hard safety problems toward easier QA problems.
• Testable: Monitor product mix over time -- if safety testing revenue shrinks relative to reliability/QA revenue, the mission drift is real.
• If wrong: Haize becomes a generic AI testing company that happens to have started with a safety narrative.

Assumption 4: The cat-and-mouse dynamic between attacks and defenses is net positive for safety.

• Evidence for: Constitutional Classifiers (which Haize helped test) reduced jailbreak success from 86% to 4.4%. This is a concrete defensive improvement.
• Evidence against: Bijection learning demonstrates that scaling creates new attack surfaces. Published attacks (bijection learning is open-source) also empower malicious actors. The attack surface is exponentially large.
• Testable: Track whether the fraction of harmful outputs in deployed models decreases over time despite increasing capabilities.
• If wrong: Automated red-teaming accelerates the arms race without net safety improvement.

Strengths

Genuine technical capability. The team produces real research -- bijection learning at ICLR, the ACG speedup, Cascade, Verdict. These are not marketing claims; they're peer-reviewed contributions that others cite and build on.

The inverse scaling finding is genuinely important. Demonstrating that more capable models are more vulnerable to bijection learning directly challenges the "capabilities-safety correlation" narrative. This finding, if it generalizes, has deep implications for AI safety strategy.

Commercial sustainability. Unlike nonprofit safety orgs dependent on a few philanthropic funders, Haize has a viable business model. Multi-million dollar contracts and a $100M valuation suggest market validation. Commercial sustainability means they can operate independently of the goodwill of any single funder.

Practical impact on defenses. The Constitutional Classifiers collaboration produced a concrete 95%+ reduction in jailbreak success. This is a measurable defensive contribution.

Low compute dependency. Unlike frontier labs or even some safety research orgs, Haize doesn't depend on gifted compute. Their work requires API access, not H100 clusters. This preserves some independence.

Young, fast, research-oriented. The founding team (all ~24, all Harvard CS, multiple with ML research backgrounds) has published 15+ papers despite the company being under 2 years old. The research-first culture, if maintained, is a strength.

Weaknesses and Risks

The conflict of interest is structural, not incidental. Being paid by the companies you evaluate creates an incentive to produce results that maintain the relationship. No amount of good intentions eliminates this structural problem. True independence requires funding from a third party (government, consortium, or philanthropy). Haize chose the VC path, which trades independence for sustainability.

No mechanism for action on findings. Tang admits they don't know how their findings are used. They have no whistleblower protocol, no disclosure mechanism, and no power to delay or prevent deployment. At best, they provide information; at worst, they provide cover. The decision to act on findings rests entirely with the labs.

The "security theater" critique is not addressed. If red-teaming has never blocked a model release (per the academic survey), what evidence is there that Haize's work changes outcomes? The Constitutional Classifiers collaboration shows defensive improvement, but Haize is a testing company, not a defense company. They find problems; they don't control whether problems are fixed.

Mission drift is already visible. The pivot from "Bad day to be a language model" to "Deploy 99.9% Reliable AI" is a move from safety toward enterprise reliability. The product suite expansion (Monitor, Robustify, Sphynx) extends beyond red-teaming into general AI ops. The enterprise client base (Deloitte, MongoDB) has different needs than frontier labs. As commercial pressures increase, the safety mission may become secondary.

Zero engagement with the safety research community. No forum posts, no Coefficient Giving grants, no involvement with EA/rationalist/safety networks. This means Haize is not receiving feedback from or contributing to the intellectual community most focused on AI catastrophic risk. They may be solving the wrong problems at scale.

NDAs limit impact. The most important findings are the ones they can't talk about. NDAs covering vulnerability specifics mean the public (and other researchers) cannot learn from Haize's most significant work.

Tiny team, concentrated risk. ~10-14 employees for a $100M company means enormous key-person risk. If Tang left or the founding team fractured, the company could collapse.

Cross-References

METR and Apollo Research are the closest comparison. Both provide third-party model evaluation, but both are nonprofits with different incentive structures. METR focuses on dangerous capability evaluations (CBRN, autonomy); Apollo focuses on deception and scheming. Haize focuses on automated adversarial robustness. They are complementary rather than competing -- Haize does breadth testing, METR/Apollo do depth testing on the most dangerous capabilities.

Gray Swan (Andy Zou, Zico Kolter) is the closest competitor. They build adversarially robust models rather than testing others' models. Haize tested against Gray Swan models and benchmarks against them. Tang and Zou collaborated at Harvard. There is both competition and collaboration.

Robust Intelligence (acquired for ~$100M in 2023) is the commercial competitor Haize is most often compared to. They focus on AI security rather than red-teaming specifically.

UK AISI and NIST represent the government-led evaluation model that could either complement or displace commercial red-teaming. Tang has been skeptical of NIST's concreteness. Haize does not appear to have direct government partnerships.

Anthropic is both client and collaborator. The Constitutional Classifiers work is Haize's highest-profile contribution. The relationship is symbiotic but asymmetric -- Anthropic controls what is published and how findings are used.

What Would Change This Assessment

• Evidence that Haize findings delayed or prevented a model release. This would validate the theory of change concretely.
• Adoption of a whistleblower/disclosure protocol. This would address the governance gap.
• Tang speaking candidly about existential risk. If the "Crystallizing AI Risk" blog post reveals genuine concern about catastrophic AI risk, it would suggest the commercial framing is strategic rather than reflecting the founders' actual motivations.
• Transition to a government or consortium funding model. If Haize secured funding from a source other than VCs and client contracts, the independence problem would be partially addressed.
• Evidence of mission drift -- if safety testing revenue drops below 50% of total revenue, the reliability pivot has won.
• A high-profile failure -- if a model Haize tested causes significant harm after deployment, it would test whether Haize had been thorough or providing cover.

Self-Critique

What sources should I have checked but didn't?

• Tang's "Crystallizing AI Risk" blog post (inaccessible, JS-rendered). This is the biggest gap -- it could significantly change the assessment of his motivations.
• The VentureBeat founding article (rate-limited). An early profile might reveal initial positioning different from current messaging.
• Haize's actual website and blog (both blocked). The official self-description is not in the evidence base.
• The AI Security Podcast full transcript (partially read). The commercial positioning details may be more revealing than the catalog summary suggests.

Where is this analysis potentially biased?

• I may be applying too harsh a standard for "independence." Most safety evaluators have some relationship with the entities they evaluate. The question is whether Haize's commercial relationship is worse than, say, METR's grant-funded relationship with labs that provide model access.
• I may be undervaluing the near-term contributions. Reducing jailbreak success from 86% to 4.4% (Constitutional Classifiers) is concretely useful even if it doesn't address existential risk.
• The absence of EA/LW presence is being read as a negative signal, but Haize could be legitimately pursuing safety through a commercial path that simply doesn't intersect with those communities.

What would a thoughtful person who disagrees say?

• "Haize is doing something that actually works commercially, which makes it sustainable. Nonprofit safety orgs burn through grants and produce papers nobody reads. Haize produces tools that labs actually use. That's more impactful than forum posts."
• "The independence problem is overstated. Auditing firms are paid by the companies they audit, and this model works well enough in finance. What matters is reputational skin in the game."
• "The inverse scaling finding alone justifies Haize's existence. This is a genuinely important safety result that came from a for-profit red-teaming company."

What's my single weakest claim? The claim that Haize's work may be "security theater" is the weakest, because the Constitutional Classifiers collaboration demonstrates concrete defensive value. The security theater critique applies more to the field's overall self-presentation than to Haize's specific technical contributions.

What information would most change my view? Evidence that Haize's findings have directly caused a lab to delay or modify a model release. This would validate the entire theory of change.