Gray Swan AI

Stated Theory of Change

Gray Swan's stated theory of change is enterprise-focused: AI systems are vulnerable, enterprises need to deploy AI securely, Gray Swan provides the tools (Cygnal filtering, Shade testing) and threat intelligence (Arena competitions) to enable secure deployment. The company positions itself as "the safety and security provider for the AI era."

The causal chain: Gray Swan discovers vulnerabilities through research and 12,000+ Arena red-teamers -> threat intelligence feeds into defensive products -> enterprises deploy these products -> AI deployments become more secure -> reduced harm from AI misuse in production.

This is a cybersecurity theory of change, not an AI safety theory of change. It addresses the question "how do we prevent jailbreaking and misuse of deployed AI systems?" rather than "how do we ensure increasingly powerful AI systems remain aligned with human values?"

Revealed Theory of Change

Gray Swan's actions reveal a more nuanced theory:

Offense drives defense. The company's deepest conviction, evident in the research trajectory (GCG -> HarmBench -> Circuit Breakers -> Arena), is that you can only defend AI systems by first understanding how to attack them. This is the classical cybersecurity "red team" philosophy applied to AI. The Arena makes this philosophy into a business: crowdsource attacks, then sell defenses.

Research is the moat. Gray Swan's actual competitive advantage is not its products but its research lineage. GCG (400+ citations), Circuit Breakers (NeurIPS), HarmBench (adopted by national AI Safety Institutes), RepE, WMDP -- this body of work, produced primarily at CMU and CAIS before Gray Swan existed, gives the company credibility that purely commercial competitors lack. The revealed strategy is: maintain research leadership to attract lab partnerships and government sponsorship, then convert credibility into commercial products.

The Arena is the flywheel. The Arena simultaneously generates: threat intelligence data (1.8M attacks), marketing credibility (UK AISI and US AISI co-judging), a hiring pipeline (10+ contractors, VP-level hires), benchmarking data labs want, and a community of 12,000 red-teamers who provide ongoing attack innovation for free. This is a genuinely clever business construction.

Divergence from stated theory. The stated theory is about enterprise security. The revealed actions show something more hybrid: (a) genuine safety research that advances the field, (b) government partnerships that position Gray Swan close to the regulatory apparatus, and (c) commercial products that pay the bills. The tension is that (a) and (b) point toward a public-good mission while (c) requires commercial optimization.

The CAIS-Gray Swan pipeline. Research funded by philanthropy (Open Philanthropy, FTX) through CAIS, and by public universities (CMU), was subsequently commercialized by Gray Swan. This is a standard academic-to-startup pipeline, but the nonprofit intermediary adds a layer: CAIS's mission was reducing AI risk, and Gray Swan commercializes that work for enterprise customers. Whether this is mission-aligned (commercializing safety research makes it more widely used) or mission-diluting (commercial incentives reshape what gets built) depends on execution.

Key Assumptions

1. Adversarial testing produces genuine safety improvements, not just security theater.

• Evidence for: The Constitutional Classifiers collaboration (Haize + Gray Swan + UK AISI + Anthropic) reduced jailbreak success from 86% to 4.4%. Arena data provides concrete comparative ASR metrics. Labs pay for this testing.
• Evidence against: The "security theater" paper finds no case where red-teaming prevented a model release. Gray Swan has no mechanism to compel labs to act on findings. Nick Winter (Gray Swan VP) observed "no clear trend towards newer models being more secure."
• Testable: Track whether models tested through Gray Swan's Arena show measurably improved safety in subsequent versions.
• If wrong: Gray Swan provides expensive reassurance rather than genuine risk reduction -- a dangerous outcome if enterprises believe they are protected when they are not.

2. Circuit Breakers are a viable defense mechanism.

• Evidence for: NeurIPS 2024 acceptance. Survived the full month of the Jailbreaking Championship (2 of 3 cygnet models unbroken). Low added latency.
• Evidence against: TU Munich showed 100% bypass with three simple modifications and no hyperparameter tuning. Confirm Labs found 39% over-refusal rate on OR-Bench. The safety-helpfulness tradeoff is real and substantial.
• Testable: Whether Gray Swan has updated circuit breakers to resist the TU Munich attack (no public evidence as of March 2026).
• If wrong: Gray Swan's core technical contribution is a defensive technique that does not survive serious adversarial pressure, and their product claims are overstated.

3. The Arena community model is sustainable and scalable.

• Evidence for: 12,000+ participants, $370K+ paid out, government co-judging, lab sponsorship, VP-level hire from Arena.
• Evidence against: Arena operations are expensive (prize pools, hosting, moderation). Lab sponsorship may be contingent on continued novelty. Competitors (XBOW, Armadin) have vastly more capital to build competing platforms.
• Testable: Whether Arena participation grows and retains top talent against better-funded alternatives.
• If wrong: Gray Swan loses its primary data advantage and hiring pipeline.

4. A $5.68M seed is sufficient to compete in a market where competitors raised $120M-$190M.

• Evidence for: Gray Swan has a research pedigree and government relationships that money cannot buy quickly. Arena gives them proprietary data. Academic connections reduce labor costs (PhD students, postdocs).
• Evidence against: Enterprise sales requires investment in go-to-market, customer success, and product engineering. The CSO hire signals this need. Without more capital, Gray Swan may be outcompeted on execution even with better research.
• Testable: Whether a Series A materializes in 2026.
• If wrong: Gray Swan is acquired, runs out of money, or becomes a research shop that cannot compete commercially.

Strengths

1. Research pedigree is unmatched among AI security startups. GCG (400+ citations), Circuit Breakers (NeurIPS), HarmBench (adopted by AI Safety Institutes), WMDP, RepE, Safety Pretraining (NeurIPS 2025), ARTEMIS. No competitor has this depth of peer-reviewed AI safety research. This gives Gray Swan unique credibility with both labs and governments.

2. Government relationships are strategically valuable. UK AISI and US AISI co-judging Arena competitions, UK government contract, OpenAI/Anthropic/DeepMind sponsorship. Gray Swan is positioned as the neutral testing ground that governments trust -- a powerful position if AI safety testing becomes mandatory.

3. The Arena is a genuinely novel competitive asset. No other AI security company has built a 12,000-person community of red-teamers generating continuous threat intelligence. The Arena produces data, talent, credibility, and benchmarks simultaneously. It is hard to replicate quickly even with more capital.

4. Dual CMU positions provide talent access. Three founders with active CMU positions can attract PhD students, postdocs, and researchers. The Safety Pretraining paper (10 authors across CMU, CAIS, Gray Swan) shows this pipeline in action.

5. The founders are technically excellent. Zou's publication record (GCG, RepE, Circuit Breakers) is extraordinary for a researcher under 30. Fredrikson's formal methods background adds rigor that pure ML researchers lack. Kolter's stature opens doors at the highest levels.

Weaknesses and Risks

1. The Kolter conflict is a ticking time bomb. Kolter chairs OpenAI's Safety Committee (can halt model releases) while co-founding a company that sells safety testing tools. If this undisclosed conflict becomes public in the way the Hendrycks/SB-1047 conflict did, it could be far more damaging because Kolter holds actual governance power. The absence of a disclosed conflict-of-interest policy makes this worse, not better.

2. Circuit Breakers may be fundamentally limited. The TU Munich 100% bypass, the 39% over-refusal rate on OR-Bench, and the Jailbreaking Championship's observation that safety training degrades helpfulness all point to the same issue: circuit breakers trade capability for safety in a way that may not be acceptable to enterprise customers. Gray Swan's commercial pitch ("innovate at full speed without compromising security") may be overpromising.

3. Severe underfunding relative to competition. $5.68M vs. $120M-$190M for competitors. In a market requiring enterprise sales infrastructure, this gap is dangerous. Gray Swan's research advantage may not matter if competitors ship faster and capture enterprise contracts.

4. No evidence of downstream safety impact. Despite generating 62K breaches in the UK AISI challenge and running continuous Arena operations, there is no documented case of Gray Swan's findings causing a lab to delay, modify, or not release a model. The same structural problem that plagues the entire commercial red-teaming industry applies to Gray Swan.

5. The CAIS-Gray Swan IP pipeline raises integrity questions. Research funded by philanthropy (Open Phil's $12.5M to CAIS, FTX's $6.5M) was commercialized at a for-profit startup co-founded by the same people. This is legal and common in academia, but it creates a perception problem: donors who funded CAIS to reduce AI risk may not have expected the research to become a for-profit product.

6. Zero engagement with the AI safety research community. No forum posts, no community discourse, no philanthropic funding. Gray Swan exists entirely in the cybersecurity/enterprise market. This means they are not accountable to or informed by the intellectual community most focused on catastrophic AI risk.

Cross-References

Gray Swan vs. Haize Labs: The most direct competitor. Both do third-party AI red-teaming. Key differences: Gray Swan has the research pedigree (GCG, Circuit Breakers), Haize has more capital ($12.5M vs. $5.68M) and a stronger enterprise reliability pitch. Gray Swan builds defensive technology (circuit breakers, Cygnal); Haize focuses on offensive testing. Leonard Tang (Haize CEO) collaborated with Andy Zou at Harvard. The inverse scaling finding (Haize) and the circuit breaker technique (Gray Swan) address the same problem from opposite sides.

Gray Swan vs. CAIS: Same founding team, different legal structures. CAIS is the nonprofit producing research and policy advocacy (MAIM, Superintelligence Strategy). Gray Swan is the for-profit commercializing that research. Hendrycks connects both. The question is whether they are complementary (CAIS does research, Gray Swan does products) or whether the for-profit's commercial needs distort the nonprofit's priorities.

Gray Swan vs. XBOW/Armadin/RunSybil: Pure cybersecurity competitors with 20-30x more capital. They focus on general software vulnerabilities; Gray Swan specializes in AI-specific attacks. Gray Swan's research moat is real but may not survive a well-funded competitor building an equivalent platform.

Gray Swan and OpenAI: Kolter chairs OpenAI's Safety Committee AND co-founded Gray Swan. This is not just a cross-reference -- it is a governance entanglement. OpenAI is both Gray Swan's regulator (via Kolter's committee role) and its customer/sponsor (via Arena participation).

What Would Change This Assessment

• A disclosed conflict-of-interest policy for Kolter's dual role would significantly reduce governance concerns.
• A published response to the TU Munich circuit breaker bypass would clarify whether the core technology has evolved.
• A Series A raise would signal commercial viability and investor confidence.
• Documented evidence that Arena findings changed a lab's deployment decision would validate the entire theory of change.
• An interview where Fredrikson or Zou articulate their personal theory of change would clarify whether Gray Swan is a safety company doing enterprise or an enterprise company doing safety.
• Open Phil or SFF investing in Gray Swan would signal alignment with the broader AI safety ecosystem.

Self-Critique

Sources I should have checked but did not: The PGH Tech podcast with founders (audio only, no transcript). The 20MinVC episode with Kolter (audio only). These would likely contain the most candid articulations of the founders' actual vision. Also: OpenAI's formal announcement of Kolter (failed to fetch) might contain conflict-of-interest disclosures.

Potential bias: I may be overweighting the governance concerns because the Pirate Wires article is vivid and well-documented, while the positive governance story (Kolter using his OpenAI position to genuinely improve safety) is by nature private and undocumented. The circuit breaker bypass critique may be overstated -- white-box embedding attacks are an unrealistically strong threat model for production systems (as the TU Munich paper itself acknowledges).

A thoughtful person who disagrees would say: "Gray Swan is doing exactly what the AI safety field needs -- taking academic research and making it commercially viable. The Arena is the best public evaluation of AI model safety that exists. The circuit breaker bypass requires white-box access that real attackers don't have. The Kolter situation is normal dual-role stuff in academia. And being VC-funded is actually better than being grant-funded because it forces discipline and sustainability."

Weakest claim: That the CAIS-Gray Swan IP pipeline is problematic. Academic-to-startup technology transfer is how the US innovation system works. The founders had every right to commercialize their research. The question is whether the nonprofit's donors understood this would happen -- and that question may have an innocent answer.

Information that would most change my view: (1) Evidence that Kolter's Safety Committee has actually used its authority to delay an OpenAI model release, which would demonstrate that the dual role produces genuine safety value. (2) A candid interview with Fredrikson or Zou explaining their theory of change for AI risk reduction, which would clarify whether the enterprise framing is strategic or reflects their actual beliefs.