Co-Executive Director at ML Alignment & Theory Scholars Program (2022-present)
Co-Founder & Board Member at London Initiative for Safe AI (2023-present)
Manifund Regrantor (2023-present) | RFPs here
Advisor, Catalyze Impact (2023-present) | ToC here
Advisor, AI Safety ANZ (2024-present)
Ph.D. in Physics at the University of Queensland (2017-2023)
Group organizer at Effective Altruism UQ (2018-2021)
Give me feedback! :)
I analyzed the research interests of the 454 Action Editors on the Transactions on Machine Learning Research (TMLR) Editorial Board to determine what proportion of ML academics are interested in AI safety (credit to @scasper for the idea).
14% of editors listed any of “robustness”, “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest.
10% of editors listed any of “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest. I excluded “robustness” as much of this research is considered more “capabilities” than “safety”.
3.7% of editors listed “safety” or “alignment” as a research interest.
a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence
Sounds like PIBBSS/PrincInt!
Cheers, Cleo!
Even if AI safety founders should be risk averse, I think we should do better at supporting the relatively few competent founder-types who are deeply interested in AI safety.
I suspect that we disagree significantly on the potential downside risk of most AI safety startups. I think it’s relatively hard to have a significant negative impact, particularly one that outweighs the expected benefits, given how much optimization pressure is being applied to advancing AI capabilities across the economy. Creating a new frontier AI company (e.g., Mistral-sized) or a toxic advocacy org would be notable exception. Maybe Mechanize and Calaveras are exceptions too?
Note that at least Anthropic has a hard time finding talent that is also mission-aligned, which they prefer, particularly for safety teams.
Thanks for sharing, Akshyae! Based on the DMs I received after posting this, I think your experience is unfortunately common. Great job sticking at it and launching Explainable!
Can you share which?
True, but we accepted 75% of all scholars into the 6-month extension last program, so the pressure might not be that large now.
Of note: when I first approached you about becoming a MATS mentor, I don’t think you had significant field-building or mentorship experience and had relatively few papers. Since then, you have become one of the most impactful field-builders, mentors, and researchers in AI safety, by my estimation! This is a bet I would take again.
Edit: I mistakenly said “27% at frontier labs” when I should have said “27% at for-profit companies”. Also, note that this is 27% of those working on AI safety (80%), so 22% of all alumni.
In regards to adversarial selection, we can compare MATS to SPAR. SPAR accepted ~300 applicants in their latest batch, ~3x MATS (it’s easier to scale if you’re remote, don’t offer stipends, and allow part-timers). I would bet that the average research impact of SPAR participants is significantly lower than that of MATS, though there might be plenty of confounders here. It might be worth doing a longitudinal study here comparing various training programs’ outcomes over time, including PIBBSS, ERA, etc.
I think your read of the situation re. mentor ratings is basically correct: increasingly many MATS mentors primarily care about research execution ability (generally ML), not AI safety strategy knowledge. I see this as a feature, not a bug, but I understand why you disagree. I think you are prioritizing a different skillset than most mentors that our mentor selection committee rates highly. Interestingly, most of the technical mentors that you rate highly seem to primarily care about object-level research ability and think that strategy/research taste can be learned on the job!
Note that I think the pendulum might start to swing back towards mentors valuing high-level AI safety strategy knowledge as the Iterator archetype is increasingly replaced/supplemented by AI. The Amplifier archetype seems increasingly in-demand as orgs scale, and we might see a surge in Connectors as AI agents improve to the point that their theoretical ideas are more testable. Also note that we might have different opinions on the optimal ratio of “visionaries” vs. “experimenters” in an emerging research field.
I like this comment. I think it’s easy to overfit on the most salient research agendas, especially if there are echo chambers and tight coupling between highly paid frontier AI staff and nonprofit funders. The best way I know to combat this at MATS is:
Maintain a broad church of AI safety research, including deliberately making mentor “diversity picks” and choosing a mentor selection committee that contains divergent thinkers. As another example, I think Constellation has done a good job recently at expanding member diversity and reducing echo chambers.
Requiring that COIs be declared and mitigated, including along reporting chains, at the same organization, with romantic/sexual partners, and with frequent research collaborators.
Encouraging “scout mindset” and “reasoning transparency”, especially among people with divergent beliefs. I think this is a large strength of MATS: we are a melting pot for ideas and biases.
Note that I expect overfitting to decrease with further scale and diversity, given the above practices are adhered to!
I’m sorry to hear about your paradoxical impact; this sounds tough and it’s a fear I share. I feel a bit better about MATS’ impact because very few of our alumni work on AI capabilities at frontier labs (~2% by my estimation) and very few work at OpenAI altogether, but I can understand if you feel that the 22% working on AI safety at for-profit companies are primarily doing “safetywashing” or something (on net I disagree, but it’s a valid concern).
I think there is something for me to learn from your experience: at the time MIRI was running AIRCS, OpenAI was not an AI safety pariah; it’s possible that some of the companies that MATS alums join now will become pariahs in future, revealing paradoxical impact. I’m not sure what to do about this other than encourage people to be intentional with their careers, question assumptions, and “don’t do evil” (the MATS values are impact first, scout mindset, reasoning transparency, and servant leadership). I think that AI safety has to scale to have a chance at solving alignment in time; this means that some people will end up working on counter-productive things. I can understand if your risk tolerance is different than mine, or you are more skeptical about the impact of MATS or the founders who might be inspired by my post.
You’re probably right that the best startups come from people who have great experience in the thing, but plenty of profitable startups get founded by kids out of college. The risk/reward tradeoff is probably different in tech. I think the best AI safety field-building startups were founded/scaled by people with experience in fieldbuilding (e.g., my experience with an EA UQ, Dewi’s experience with EA Cambridge, Agus’ experience with CEA, etc.), but the bar might be surprisingly low.
Maybe so! I don’t think Eric Ho’s ideas are terrible and I’ve seen for-profit AI safety startups that I like (e.g., Goodfire) and that I don’t like (e.g., Softmax, probably).
Re. “useless but easy problems”, we agree to disagree. Mentor selection at MATS is very hard, so we defer a lot to a committee of experts. Admittedly, choosing this committee necessarily entails some bias. I’d be interested if anyone wants to DM me nominations!
To clarify, by “kicking out” Jeremy is referring to two mentors in particular, both of whom got a lot of support from PIBBSS and one of whom seemed to want more of an engineering assistant than a research scholar. I think both do important research and it was a tough decision, informed by our mentor selection committee, which included experts in their field, and past scholar feedback. I offered help hiring to both, including our alumni hiring database.
No worries! I think research startups should be founded by strong researchers. But there are lots of potentially impactful startups (field-building, advocacy, product, etc.) that don’t require founders with research skills, and these might be best served by learning on the job?
The former. Even large research nonprofits (e.g., RAND, AI2, ATI, SFI) tend to be led by people with research experience, though they probably do a lot less research than CEOs at small research orgs.
I completely agree with you! Where did you think I implied the opposite?
I think you are probably unusually good at spotting which mech interp orgs are doomed ex ante, but you aren’t infallible. And I think a situation where many small startups are being founded, even if most will be doomed, is what a functional startup ecosystem looks like! We don’t want people working on obviously bad ideas, but I naively expect the process of startup ideation and experimentation, aided by VC money, to yield good mech interp directions.
Gemini 3 estimates that there are 15-20k core ML academics and 100-150k supporting PhD students and Postdocs worldwide. If the TMLR sample is representative, this indicates that there are:
~20k academics interested in any of the above research areas.
~15k academics interested in the non-robustness research areas.
~5k academics interested in AI safety or alignment (note that this might include RLHF).