TL;DR: In AI safety, we systematically undervalue founders and field‑builders relative to researchers and prolific writers. This status gradient pushes talented would‑be founders and amplifiers out of the ecosystem, slows the growth of research orgs and talent funnels, and bottlenecks our capacity to scale the AI safety field. We should deliberately raise the status of founders and field-builders and lower the friction for starting and scaling new AI safety orgs.
Epistemic status: A lot of hot takes with less substantiation than I’d like. Also, there is an obvious COI in that I am an AI safety org founder and field-builder.
Coauthored with ChatGPT.
Why boost AI safety founders?
Multiplier effects: Great founders and field-builders have multiplier effects on recruiting, training, and deploying talent to work on AI safety. At MATS, mentor applications are increasing 1.5x/year and scholar applications are increasing even faster, but deployed research talent is only increasing at 1.25x/year. If we want to 10-100x the AI safety field in the next 8 years, we need multiplicative capacity, not just marginal hires; training programs and founders are the primary constraints.
Anti-correlated attributes: “Founder‑mode” is somewhat anti‑natural to “AI concern.” The cognitive style most attuned to AI catastrophic risk (skeptical, risk‑averse, theory-focused) is not the same style that woos VCs, launches companies, and ships MVPs. If we want AI safety founders, we need to counterweight the selection against risk-tolerant cognitive styles to prevent talent drift and attract more founder-types to AI safety.
Adverse incentives: The dominant incentive gradients in AI safety point away from founder roles. Higher social status, higher compensation, and better office/advisor access often accrue to research roles, so the local optimum is “be a researcher,” not “found something.” Many successful AI safety founders work in research-heavy roles (e.g., Buck Shlegeris, Beth Barnes, Adam Gleave, Dan Hendrycks, Marius Hobbhahn, Owain Evans, Ben Garfinkel, Eliezer Yudkowsky) and the status ladder seems to reward technical prestige over building infrastructure. In mainstream tech, founders are much higher status than in AI safety, and e/accs vs. AI safers are arguably in competition for VC resources and public opinion.
Founder effects: AI safety (or at least security) seems on the verge of becoming mainstream and the AI safety ecosystem should capture resources or let worse alternatives flourish. Unlikely allies, including MAGA (e.g., Steve Bannon, Marjorie Taylor-Greene), the child-safety lobby, and Encode AI, recently banded together to defeat Ted Cruz’s proposed 10-year moratorium on state AI legislation. Opinionpolls indicate AI safety is a growing public concern. Many VC-backed AI security startups have launched this year (e.g., AISLE, Theorem, Virtue AI, Lucid Computing, TamperSec, Ulyssean), including via YC. We have the chance to steer public interest and capital towards greater impact, but only if we can recruit and deploy founders fast enough.
How did we get here?
Academic roots: The founders of Effective Altruism and Rationalism, the movements that popularized AI safety, were largely academics and individual contributors working in tech, not founders and movement builders. Longtermist EA and Rationalist cultures generally reward epistemic rigor, moral scrupulosity, and “lone genius” technical contributions more than building companies, shipping products, and coordinating people. Rationalists valorize “wizard power”, like making original research contributions, over “king power”, like raising and marshaling armies of researchers to solve AI alignment.
Biased spotlights: AI safety ecosystem spotlights like 80,000 Hours selectively amplify researchers and academics over founders. When AI safety founders are featured on the 80,000 Hours Podcast, they are almost always in research-heavy roles. Significant AI safety field-building orgs (e.g., BlueDot, MATS, Constellation, LISA, PIBBSS, ARENA, ERA, Apart, Pivotal) or less-influential research orgs (e.g., Apollo, EleutherAI, Goodfire, Timaeus) are generally not given much attention. The 80,000 Hours career review on “Founder of new projects tackling top problems” feels like a stub. Open Philanthropy RFPs technically support funding for new organizations, but this feels overshadowed by the focus on individual contributors in their branding.
Growth-aversion: AI safety grantmakers have (sometimes deliberately) throttled the growth of nascent orgs. The vibe that “rapid org scaling is risky” makes founding feel counter‑cultural. Throttling orgs can be correct in specific cases, but it generally creates a disincentive towards building by reducing confidence in grantmaker support for ambitious projects. An influential memo from 2022 argued against “mass movement building” in AI safety on the grounds that it would dilute the quality of the field; subsequently, frontier AI companies grew 2-3x/year, apparently unconcerned by dilution. Training programs (e.g., BlueDot, MATS, ARENA) and incubators (e.g., Catalyze Impact, Seldon Lab, Constellation Incubator) arrived late relative to need; even now, they occupy relatively low status positions relative to research orgs they helped build.
We should be careful not to dilute the quality of the field by scaling too fast. I confess that I don’t really understand this concern. If outreach funnels attract a large number of low-caliber talent to AI safety, we can enforce high standards for research grants and second-stage programs like ARENA and MATS. If forums like LessWrong or the EA Forum become overcrowded with low-calibre posts, we can adjust content moderation or the effect of karma on visibility. As a last resort, field growth could be scaled back via throttled grant funding. Additionally, growing the AI safety field is far from guaranteed to reduce the average quality of research, as most smart people are not working on AI safety and, until recently, AI safety had poor academic legibility. Even if growing the field reduces the average researcher quality, I expect this will result in more net impact.
Great founders don’t need help/coddling; they make things happen regardless. While many great founders succeed in the absence of incubators or generous starting capital, Y Combinator has produced some great startups! Adding further resources to aid founders seems unlikely to be negative value and will likely help potential founders who lack access to high-value spaces like Constellation, LISA, or FAR Labs, which are frequented by grantmakers and AI safety tastemakers. As an example, if not for Lightcone Infrastructure’s Icecone workshop in Dec 2021-Jan 2022, I would probably have found it hard to make the necessary connections and positive impressions to help MATS scale.
What should we do?
Narrative shift: Prominent podcasts like 80,000 Hours should publish more interviews with AI safety founders and field-builders. Someone should launch an “AI safety founders” podcast/newsletter that spotlights top founders and their journeys.
Career surfaces: Career advisors like 80,000 Hours and Probably Good should make “AI safety org founder” and “AI safety field-builder” a first‑class career path in guides and advising. Incubators like Halcyon Futures, Catalyze Impact, Seldon Lab, Constellation Incubator, etc. should be given prominence.
Capital surfaces: Funders like Open Philanthropy should launch RFPs explicitly targeted towards new org formation with examples of high-impact projects they want founded.
Social surfaces: AI safety hubs like Constellation, LISA, FAR Labs, and Mox should host events for aspiring founders. Field-building programs should launch founders networks to provide warm intros to mentors/advisors, grantmakers/VCs, and fiscal sponsorship orgs.
AI safety undervalues founders
TL;DR: In AI safety, we systematically undervalue founders and field‑builders relative to researchers and prolific writers. This status gradient pushes talented would‑be founders and amplifiers out of the ecosystem, slows the growth of research orgs and talent funnels, and bottlenecks our capacity to scale the AI safety field. We should deliberately raise the status of founders and field-builders and lower the friction for starting and scaling new AI safety orgs.
Epistemic status: A lot of hot takes with less substantiation than I’d like. Also, there is an obvious COI in that I am an AI safety org founder and field-builder.
Coauthored with ChatGPT.
Why boost AI safety founders?
Multiplier effects: Great founders and field-builders have multiplier effects on recruiting, training, and deploying talent to work on AI safety. At MATS, mentor applications are increasing 1.5x/year and scholar applications are increasing even faster, but deployed research talent is only increasing at 1.25x/year. If we want to 10-100x the AI safety field in the next 8 years, we need multiplicative capacity, not just marginal hires; training programs and founders are the primary constraints.
Anti-correlated attributes: “Founder‑mode” is somewhat anti‑natural to “AI concern.” The cognitive style most attuned to AI catastrophic risk (skeptical, risk‑averse, theory-focused) is not the same style that woos VCs, launches companies, and ships MVPs. If we want AI safety founders, we need to counterweight the selection against risk-tolerant cognitive styles to prevent talent drift and attract more founder-types to AI safety.
Adverse incentives: The dominant incentive gradients in AI safety point away from founder roles. Higher social status, higher compensation, and better office/advisor access often accrue to research roles, so the local optimum is “be a researcher,” not “found something.” Many successful AI safety founders work in research-heavy roles (e.g., Buck Shlegeris, Beth Barnes, Adam Gleave, Dan Hendrycks, Marius Hobbhahn, Owain Evans, Ben Garfinkel, Eliezer Yudkowsky) and the status ladder seems to reward technical prestige over building infrastructure. In mainstream tech, founders are much higher status than in AI safety, and e/accs vs. AI safers are arguably in competition for VC resources and public opinion.
Founder effects: AI safety (or at least security) seems on the verge of becoming mainstream and the AI safety ecosystem should capture resources or let worse alternatives flourish. Unlikely allies, including MAGA (e.g., Steve Bannon, Marjorie Taylor-Greene), the child-safety lobby, and Encode AI, recently banded together to defeat Ted Cruz’s proposed 10-year moratorium on state AI legislation. Opinion polls indicate AI safety is a growing public concern. Many VC-backed AI security startups have launched this year (e.g., AISLE, Theorem, Virtue AI, Lucid Computing, TamperSec, Ulyssean), including via YC. We have the chance to steer public interest and capital towards greater impact, but only if we can recruit and deploy founders fast enough.
How did we get here?
Academic roots: The founders of Effective Altruism and Rationalism, the movements that popularized AI safety, were largely academics and individual contributors working in tech, not founders and movement builders. Longtermist EA and Rationalist cultures generally reward epistemic rigor, moral scrupulosity, and “lone genius” technical contributions more than building companies, shipping products, and coordinating people. Rationalists valorize “wizard power”, like making original research contributions, over “king power”, like raising and marshaling armies of researchers to solve AI alignment.
Biased spotlights: AI safety ecosystem spotlights like 80,000 Hours selectively amplify researchers and academics over founders. When AI safety founders are featured on the 80,000 Hours Podcast, they are almost always in research-heavy roles. Significant AI safety field-building orgs (e.g., BlueDot, MATS, Constellation, LISA, PIBBSS, ARENA, ERA, Apart, Pivotal) or less-influential research orgs (e.g., Apollo, EleutherAI, Goodfire, Timaeus) are generally not given much attention. The 80,000 Hours career review on “Founder of new projects tackling top problems” feels like a stub. Open Philanthropy RFPs technically support funding for new organizations, but this feels overshadowed by the focus on individual contributors in their branding.
Growth-aversion: AI safety grantmakers have (sometimes deliberately) throttled the growth of nascent orgs. The vibe that “rapid org scaling is risky” makes founding feel counter‑cultural. Throttling orgs can be correct in specific cases, but it generally creates a disincentive towards building by reducing confidence in grantmaker support for ambitious projects. An influential memo from 2022 argued against “mass movement building” in AI safety on the grounds that it would dilute the quality of the field; subsequently, frontier AI companies grew 2-3x/year, apparently unconcerned by dilution. Training programs (e.g., BlueDot, MATS, ARENA) and incubators (e.g., Catalyze Impact, Seldon Lab, Constellation Incubator) arrived late relative to need; even now, they occupy relatively low status positions relative to research orgs they helped build.
Potential counter-arguments
We don’t have enough good ideas to deploy talent at scale, so founders/field-builders aren’t important. I disagree; I think there are many promising AI safety research agendas that can absorb talent for high impact returns (e.g., AI control, scalable oversight, AI governance, open-weight safety, mech interp, unlearning, cooperative AI, AIXI safety, etc.). Also, if ideas are the bottleneck, a “hits-based approach” seems ideal! We should be launching more AI safety ideas bounties and contests, agenda incubators like Refine and the PIBBSS x Iliad residency, and research programs like AE Studio’s “Neglected Approaches” initiative. Most smart people are outside the AI safety ecosystem, so outreach and scaling seem critical to spawning more AI safety agendas.
We should be careful not to dilute the quality of the field by scaling too fast. I confess that I don’t really understand this concern. If outreach funnels attract a large number of low-caliber talent to AI safety, we can enforce high standards for research grants and second-stage programs like ARENA and MATS. If forums like LessWrong or the EA Forum become overcrowded with low-calibre posts, we can adjust content moderation or the effect of karma on visibility. As a last resort, field growth could be scaled back via throttled grant funding. Additionally, growing the AI safety field is far from guaranteed to reduce the average quality of research, as most smart people are not working on AI safety and, until recently, AI safety had poor academic legibility. Even if growing the field reduces the average researcher quality, I expect this will result in more net impact.
Great founders don’t need help/coddling; they make things happen regardless. While many great founders succeed in the absence of incubators or generous starting capital, Y Combinator has produced some great startups! Adding further resources to aid founders seems unlikely to be negative value and will likely help potential founders who lack access to high-value spaces like Constellation, LISA, or FAR Labs, which are frequented by grantmakers and AI safety tastemakers. As an example, if not for Lightcone Infrastructure’s Icecone workshop in Dec 2021-Jan 2022, I would probably have found it hard to make the necessary connections and positive impressions to help MATS scale.
What should we do?
Narrative shift: Prominent podcasts like 80,000 Hours should publish more interviews with AI safety founders and field-builders. Someone should launch an “AI safety founders” podcast/newsletter that spotlights top founders and their journeys.
Career surfaces: Career advisors like 80,000 Hours and Probably Good should make “AI safety org founder” and “AI safety field-builder” a first‑class career path in guides and advising. Incubators like Halcyon Futures, Catalyze Impact, Seldon Lab, Constellation Incubator, etc. should be given prominence.
Capital surfaces: Funders like Open Philanthropy should launch RFPs explicitly targeted towards new org formation with examples of high-impact projects they want founded.
Social surfaces: AI safety hubs like Constellation, LISA, FAR Labs, and Mox should host events for aspiring founders. Field-building programs should launch founders networks to provide warm intros to mentors/advisors, grantmakers/VCs, and fiscal sponsorship orgs.
How to become a founder
Apply to an incubator that understands AI safety, like Halcyon Futures, Catalyze Impact, Seldon Lab, Constellation Incubator, Fifty Years 5050 AI, Entrepeneur First def/acc. YC has also funded several AI security and interpretability startups.
Draft a 1‑page pitch or theory of change and circulate it via Slack groups, forums, office happy hours, or friends.
Join the AI Safety Founders Network and ask for feedback on your idea.
Apply to RFPs that allow org funding, like Open Philanthropy’s “Funding for work that builds capacity to address risks from transformative AI”.
Talk to VCs/angels when there’s a plausible revenue path. Brainstorm for-profit AI alignment org ideas.