Anti-correlated attributes: “Founder‑mode” is somewhat anti‑natural to “AI concern.” The cognitive style most attuned to AI catastrophic risk (skeptical, risk‑averse, theory-focused) is not the same style that woos VCs, launches companies, and ships MVPs. If we want AI safety founders, we need to counterweight the selection against risk-tolerant cognitive styles to prevent talent drift and attract more founder-types to AI safety.
I think AI safety founders should be risk-averse.
For-profit investors like risk-seeking founders because for-profit orgs have unlimited upside and limited downside (you can’t lose more money than you invest), and hence investors can expect ROI on a portfolio of high-variance, decorrelated startups. You get high variance with risk-seeking founders, and decorrelation with contrarian founders. But AI safety isn’t like this. The downside is just as unlimited as the upside, so you can’t expect ROI simply because the orgs are high-variance and uncorrelated, c.f. unilateralist curse.
An influential memo from 2022 argued against “mass movement building” in AI safety on the grounds that it would dilute the quality of the field; subsequently, frontier AI companies grew 2-3x/year, apparently unconcerned by dilution.
I think frontier labs have an easier time selecting for talent than AI safety orgs. Partly because they need to care less about virtue/mission alignment.
Even if AI safety founders should be risk averse, I think we should do better at supporting the relatively few competent founder-types who are deeply interested in AI safety.
I suspect that we disagree significantly on the potential downside risk of most AI safety startups. I think it’s relatively hard to have a significant negative impact, particularly one that outweighs the expected benefits, given how much optimization pressure is being applied to advancing AI capabilities across the economy. Creating a new frontier AI company (e.g., Mistral-sized) or a toxic advocacy org would be notable exception. Maybe Mechanize and Calaveras are exceptions too?
Note that at least Anthropic has a hard time finding talent that is also mission-aligned, which they prefer, particularly for safety teams.
Hey Ryan, nice post. Here are some thoughts.
I think AI safety founders should be risk-averse.
For-profit investors like risk-seeking founders because for-profit orgs have unlimited upside and limited downside (you can’t lose more money than you invest), and hence investors can expect ROI on a portfolio of high-variance, decorrelated startups. You get high variance with risk-seeking founders, and decorrelation with contrarian founders. But AI safety isn’t like this. The downside is just as unlimited as the upside, so you can’t expect ROI simply because the orgs are high-variance and uncorrelated, c.f. unilateralist curse.
I think frontier labs have an easier time selecting for talent than AI safety orgs. Partly because they need to care less about virtue/mission alignment.
Cheers, Cleo!
Even if AI safety founders should be risk averse, I think we should do better at supporting the relatively few competent founder-types who are deeply interested in AI safety.
I suspect that we disagree significantly on the potential downside risk of most AI safety startups. I think it’s relatively hard to have a significant negative impact, particularly one that outweighs the expected benefits, given how much optimization pressure is being applied to advancing AI capabilities across the economy. Creating a new frontier AI company (e.g., Mistral-sized) or a toxic advocacy org would be notable exception. Maybe Mechanize and Calaveras are exceptions too?
Note that at least Anthropic has a hard time finding talent that is also mission-aligned, which they prefer, particularly for safety teams.