Principles of Intelligence (PIBBSS)

Stated Theory of Change

PIBBSS's stated theory of change has three layers:

Layer 1 -- Epistemic: Future AI systems may be sufficiently different from current ones that studying present-day ML alone won't prepare us. Intelligence is a natural phenomenon that appears across substrates (brains, ecosystems, economies, cells). By studying how intelligent behavior works in nature, we can develop substrate-independent principles that apply to AI regardless of its architecture.

Layer 2 -- Talent pipeline: The AI safety field needs people with deep expertise in complex systems, neuroscience, evolutionary biology, social science, and other domains. These experts exist in academia but face career barriers to engaging with AI safety. PIBBSS bridges this gap by pairing them with alignment researchers for structured collaboration.

Layer 3 -- Institutional: The field needs an institutional home for "blue sky" research that is too speculative for AI labs and too interdisciplinary for traditional academia. PIBBSS aims to be "a Bell Labs of AI Safety" -- a place where weird, high-variance ideas get serious support and can eventually spin off into focused research programs.

The causal chain: recruit domain experts -> pair with alignment mentors -> generate cross-domain insights -> spin off promising research lines into independent organizations or lab positions -> field grows in both breadth and quality.

Revealed Theory of Change

Where actions diverge from stated theory:

The interpretability pivot: The most significant divergence is the move toward mechanistic interpretability via AMI (Ambitious Mechanistic Interpretability) and the Iliad research group's SLT focus. These are high-quality, mathematically rigorous research directions -- but they are not the "blue sky" interdisciplinary exploration that made PIBBSS distinctive. AMI is essentially a mech interp lab. The MATS director specifically flagged this: "PIBBSS might be pivoting away from higher variance blue sky research to focus on more mainstream AI interpretability. I think this would be a mistake." This pivot is likely driven by funding incentives -- funders have shifted toward applied research, and interpretability is more legible and fundable than "what can ant armies teach us about multi-agent AI."

Scaling via structure, not serendipity: The original fellowship was radically open-ended -- find brilliant people, match them with mentors, see what happens. The new themed tracks (Cooperative AI, Gradual Disempowerment) and the move to twice-yearly fellowships with a dedicated Program Director suggest a shift toward more structured, repeatable program delivery. This is good for institutional sustainability but may reduce the high-variance hits the original model was optimized for.

From field-builder to research org: The AMI division with full-time hires represents a fundamental identity shift. A field-builder hosts other people's research; a research org does its own. The question is whether PrincInt can do both, or whether in-house research will crowd out the more speculative fellowship projects.

Nora's departure: The intellectual driving force left for a government position. The remaining leadership (Lucas Teixeira) has a thin public research identity. In practice, the intellectual vision is now distributed across the board (Ammann, Zhi-Xuan, Oldenziel, Goldhaber) and the mentor network rather than concentrated in an executive.

Key Assumptions

1. Intelligence has substrate-independent principles worth discovering.

• Evidence for: Convergent evolution in biology, universality in statistical physics, information-theoretic invariants. The Simplex results (transformers representing belief state geometry predicted by computational mechanics) are direct evidence that physics-derived theory can explain ML behavior.
• Evidence against: Many proposed analogies between natural and artificial systems have been shallow or misleading (Yudkowsky's critique of biology-based timelines). Deep learning may be "its own thing" with principles that don't map well to biological intelligence.
• Testable: Yes -- each PIBBSS-style project is a test of whether a specific cross-domain transfer works.
• If wrong: PIBBSS's value reduces to a generic talent pipeline (still valuable but not distinctively so).

2. The right talent exists in academia and can be recruited.

• Evidence for: Three cohorts successfully recruited PhDs/postdocs from physics, neuroscience, philosophy, biology. The 2023 cohort was more senior (3 years more experience than 2022). The "100+ academics" post suggests growing openness to AI safety.
• Evidence against: Social sciences remain underrecruited. Academic career incentives actively discourage AI safety work. The most senior academics (professors) are most dismissive. Retention after the program is uncertain.
• Testable: Yes -- long-term tracking of fellow careers.
• If wrong: PIBBSS becomes a brief exposure program rather than a genuine talent transition mechanism.

3. Three months is enough to produce meaningful cross-domain research.

• Evidence for: Some fellows (Gabriel Weil, Adam Shai) produced genuinely significant work. The 2024 cohort's output was more publication-ready than earlier years.
• Evidence against: PIBBSS's own 2022 retrospective: only 30-50% made "interesting progress." Prosaic alignment transfer was especially hard. The MATS director notes Connector development takes "on the order of years."
• Testable: Yes -- compare output quality across cohorts.
• If wrong: The affiliate program (6-12 months) becomes the real value creator, and the fellowship is primarily a screening/exposure tool.

4. Hits-based outcomes justify the program.

• Evidence for: Simplex alone may justify a significant portion of PIBBSS's total funding. Gabriel Weil's tort law framework. The Gradual Disempowerment paper (ICML 2025). If even 1-2 fellows per cohort produce genuinely important work, the expected value is high.
• Evidence against: No rigorous counterfactual analysis exists. Would Shai have found computational mechanics for transformers without PIBBSS? Possibly -- he was already adjacent. The hits may be cases where talented people were going to produce important work regardless.
• If wrong: PIBBSS is a nice community-building exercise with less counterfactual impact than claimed.

5. The org can survive and thrive after founder departure.

• Evidence for: Board includes several intellectually strong figures. New CG grant of $1.69M suggests funder confidence in post-Nora leadership. Nora remains on board.
• Evidence against: Lucas has no public intellectual profile. The original vision was Nora's. The mentor network was built on Nora's relationships. The "embrace the weird" positioning requires charismatic intellectual leadership.
• If wrong: PIBBSS becomes a competent but generic fellowship program, losing its distinctive epistemic identity.

Strengths

1. Genuinely distinctive intellectual niche: No other AI safety org does exactly what PIBBSS does -- bringing in deep domain experts from outside CS/ML. MATS, ERA, and other fellowships train people within the alignment paradigm; PIBBSS recruits from outside it. This is valuable regardless of any individual project's success.

2. Disproportionately strong mentor network: Having Abram Demski, John Wentworth, davidad, Jan Kulveit, and Vanessa Kosoy as mentors for a tiny org is extraordinary. This network is hard to replicate.

3. Honest self-assessment culture: The retrospective publications are unusually candid about failures. The 2022 post's admission that "concrete output was insufficient" and prosaic transfer was harder than expected builds genuine credibility.

4. Proof of concept: Simplex is a real company doing real research that genuinely emerged from the PIBBSS pipeline. Adam Shai's work on computational mechanics and transformers demonstrates the interdisciplinary transfer working at its best.

5. Nimble and adaptive: The org has evolved significantly in 4 years -- from one-off summer fellowship to affiliate program, Horizon Scanning, Iliad, AMI, twice-yearly fellowships. This suggests a leadership team that learns and adjusts.

Weaknesses and Risks

1. Extreme funder concentration: Open Philanthropy provides ~77% of known funding. If CG changes priorities, PIBBSS faces existential risk. The "5-6 months runway" crisis in late 2024 shows this is not hypothetical.

2. Identity crisis between "weird ideas" and legibility: The tension between the original blue-sky positioning and the interpretability pivot is unresolved. If PIBBSS becomes another mech interp lab, it loses its distinctive value. If it stays fully blue-sky, it may not be fundable. The org hasn't articulated a clear resolution.

3. Post-founder leadership vacuum: Lucas Teixeira has no public intellectual footprint. The vision that attracted funders, mentors, and fellows was Nora's. The org needs to develop a public intellectual identity for its current leadership or it risks becoming a "fellowship factory" without a unifying intellectual vision.

4. Uncertain impact measurement: No rigorous counterfactual analysis of fellow outcomes. The "73 alumni placed" claim is plausible but unverified. Without this data, the org is making claims it can't fully support.

5. Scale mismatch with aspiration: "Bell Labs of AI Safety" with ~5 full-time staff and $2.5M lifetime funding is a dramatic mismatch. Bell Labs had thousands of researchers and a captive corporate funding stream. The aspiration is inspiring but the path from here to there is not articulated.

6. Governance gaps: Board formed only in 2024 (year 3). Multiple board members have dual roles (Timaeus, ARIA, alumnus). No public conflict-of-interest policy. No documented selection process despite funder requests.

Cross-References

• MATS: Closest comparator. Explicitly positioned as complementary: MATS trains "Iterators" on established mentors' agendas (10 weeks); PIBBSS develops "Connectors" who bridge fields (3-12 months). MATS is larger, more mainstream, and better-funded.
• ACS Research (Jan Kulveit): Deeply interconnected. ACS's "gradual disempowerment" and "AI sociology" agendas overlap with PIBBSS fellowship tracks. Jan Kulveit is both PIBBSS mentor and ACS co-founder. These appear to be sister organizations.
• Timaeus: Connected through board member Alexander Gietelink Oldenziel. Timaeus focuses on SLT; PIBBSS's Iliad program runs SLT-adjacent work. Complementary but potential for resource competition.
• Simplex: PIBBSS's most successful spinoff. Demonstrates the pipeline working. Simplex's computational mechanics approach to interpretability is exactly the kind of cross-domain transfer PIBBSS was designed to enable.
• ARIA Safeguarded AI: Nora's current employer. The "Guaranteed Safe AI" framework that davidad leads at ARIA has intellectual roots in the PIBBSS network. This connection could channel UK government resources toward PIBBSS-style research.

What Would Change This Assessment

• Lucas Teixeira gives a substantive public talk or writes a detailed vision document -- would significantly reduce the leadership-vacuum concern.
• PIBBSS publishes audited financials or a detailed annual report -- would address transparency concerns.
• A rigorous counterfactual impact study of fellowship alumni -- would validate or challenge the talent pipeline claim.
• The AMI division produces significant interpretability results -- would validate the pivot, even if it comes at the cost of blue-sky research.
• Another fellow/affiliate produces a Simplex-caliber outcome -- would strengthen the hits-based case.
• Open Philanthropy reduces or eliminates funding -- would test whether PIBBSS can survive on diversified funding.
• A credible external critic publishes a substantive argument against the interdisciplinary approach -- would test the intellectual foundations.

Self-Critique

What sources should I have checked but didn't?

• The full Manifund page with donor comments (rate-limited during scouting)
• LinkedIn profiles of all 50+ fellows to verify placement claims
• The SFF internal evaluation that led to PIBBSS not being funded in 2025
• Any private evaluations from Open Philanthropy staff

Where is this analysis potentially biased?

• I may be overweighting the intellectual framework's elegance versus its practical impact. "Epistemic pluralism" sounds impressive but could be a sophisticated justification for unfocused research.
• I may be too charitable about the Simplex success story -- attributing it heavily to PIBBSS when Shai/Riechers may have found their approach independently.
• I may be too critical of the interpretability pivot. If the field genuinely needs more mech interp capacity, PIBBSS adding it is valuable regardless of whether it's "distinctive."

What would a thoughtful person who disagrees say? "PIBBSS is a nice community-building exercise that produces a few interesting papers per year but has minimal counterfactual impact on AI safety. The interdisciplinary transfer thesis is mostly wrong -- the most impactful safety work requires deep ML expertise, not biology analogies. The few successes (Simplex, Weil) are talented people who would have found their way regardless. The org's financial fragility is a feature revealing its marginal value, not a bug."

My single weakest claim? That the post-Nora leadership can maintain the intellectual vision and mentor network that made PIBBSS distinctive. I have essentially no evidence about Lucas Teixeira's intellectual capabilities because he hasn't made them public.

What information would most change my view? A rigorous counterfactual impact study showing that PIBBSS fellows would have entered AI safety at similar rates without the program. This would turn the assessment from "valuable hits-based approach" to "nice community with modest impact."