Co-Executive Director at ML Alignment & Theory Scholars Program (2022-present)
Co-Founder & Board Member at London Initiative for Safe AI (2023-present)
Manifund Regrantor (2023-present) | RFPs here
Advisor, Catalyze Impact (2023-present) | ToC here
Advisor, AI Safety ANZ (2024-present)
Ph.D. in Physics at the University of Queensland (2017-2023)
Group organizer at Effective Altruism UQ (2018-2021)
Give me feedback! :)
This is a hard question to answer precisely, as we have changed the metrics by which we have evaluated potential mentors several times. The average quality of mentors we accept has grown each program, by my lights. I weakly think that the average quality of mentors applying has also grown, though much slower.
Update on MATS applications trends:
Mentor applications seem to be growing 2.2x/year and we accepted 20% as primary mentors for 10.0 (Summer 2026).
Fellow applications seem to be growing 1.8x/year (counting only applicants who applied to at least one mentor) and we plan to accept 5% for 10.0.
In contrast to the apparent exponential growth in AI conference attendees, the number of AI publications since 2013 has grown quadratically (data from the Standford HAI 2025 AI Index Report). Quadratic growth in publications suggests that a linearly (constantly) increasing number of researchers are producing publications at a linear (constant) rate. Extrapolating a little, this growth rate suggests there will be 3.7M cumulative publications by 2030 (since 2013).
If the AI researcher growth rate is linear, the exponential growth of AI conference attendees might be due to increased industry presence. Also, it’s possible that the attendee growth rate is also quadratic.
Open question: how fast did the field of cybersecurity grow since the launch of the internet?
Attendees at the top-four AI conferences have been growing at 1.26x/year on average. Data is from Our World in Data. I skipped 2020-2021 for all conferences and 2022 for ICLR, as these conferences were virtual due to the COVID pandemic and had increased virtual attendance.
One could infer from these growth rates that the academic field of AI is growing 1.26x/year. Interestingly, the AI safety field (including technical and governance) seems to be growing at 1.25x/year.
The top-10 most-cited papers that MATS contributed to are (all with at least 290 citations)
Representation Engineering: A Top-Down Approach to AI Transparency
Sparse autoencoders find highly interpretable features in language models
Refusal in language models is mediated by a single direction
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
The Reversal Curse: LLMs trained on” A is B” fail to learn” B is A”
Finding neurons in a haystack: Case studies with sparse probing
Compare this to the top-10 highest-karma LessWrong posts that MATS contributed to (all with over 200 karma):
Here is a plot of the annual citations received by MATS, EleutherAI, and Apart research, adjusted so they start on the same year. The three organizations are somewhat comparable, as they leverage large networks of external collaborators: MATS mentors/fellows, EleutherAI Discord, Apart sprint participants.
The EleutherAI data fits a logistic curve perfectly, asymptoting to ~18.5k citations/year. I can’t fit the others as at least 4 data points are needed to fit a logistic curve.
I made a Google Scholar page for MATS. This was inspired by @Esben Kran’s Google Scholar for Apart Research. Eleuther AI subsequently made one too. I think all AI safety organizations and research programs should consider making Google Scholar pages to better share research and track impact.
Great post! I will fund this project on Manifund.com
Gemini 3 estimates that there are 15-20k core ML academics and 100-150k supporting PhD students and Postdocs worldwide. If the TMLR sample is representative, this indicates that there are:
~20k academics interested in any of the above research areas.
~15k academics interested in the non-robustness research areas.
~5k academics interested in AI safety or alignment (note that this might include RLHF).
I analyzed the research interests of the 454 Action Editors on the Transactions on Machine Learning Research (TMLR) Editorial Board to determine what proportion of ML academics are interested in AI safety (credit to @scasper for the idea).
14% of editors listed any of “robustness”, “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest.
10% of editors listed any of “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest. I excluded “robustness” as much of this research is considered more “capabilities” than “safety”.
3.7% of editors listed “safety” or “alignment” as a research interest.
a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence
Sounds like PIBBSS/PrincInt!
Cheers, Cleo!
Even if AI safety founders should be risk averse, I think we should do better at supporting the relatively few competent founder-types who are deeply interested in AI safety.
I suspect that we disagree significantly on the potential downside risk of most AI safety startups. I think it’s relatively hard to have a significant negative impact, particularly one that outweighs the expected benefits, given how much optimization pressure is being applied to advancing AI capabilities across the economy. Creating a new frontier AI company (e.g., Mistral-sized) or a toxic advocacy org would be notable exception. Maybe Mechanize and Calaveras are exceptions too?
Note that at least Anthropic has a hard time finding talent that is also mission-aligned, which they prefer, particularly for safety teams.
Thanks for sharing, Akshyae! Based on the DMs I received after posting this, I think your experience is unfortunately common. Great job sticking at it and launching Explainable!
Can you share which?
True, but we accepted 75% of all scholars into the 6-month extension last program, so the pressure might not be that large now.
Of note: when I first approached you about becoming a MATS mentor, I don’t think you had significant field-building or mentorship experience and had relatively few papers. Since then, you have become one of the most impactful field-builders, mentors, and researchers in AI safety, by my estimation! This is a bet I would take again.
Edit: I mistakenly said “27% at frontier labs” when I should have said “27% at for-profit companies”. Also, note that this is 27% of those working on AI safety (80%), so 22% of all alumni.
In regards to adversarial selection, we can compare MATS to SPAR. SPAR accepted ~300 applicants in their latest batch, ~3x MATS (it’s easier to scale if you’re remote, don’t offer stipends, and allow part-timers). I would bet that the average research impact of SPAR participants is significantly lower than that of MATS, though there might be plenty of confounders here. It might be worth doing a longitudinal study here comparing various training programs’ outcomes over time, including PIBBSS, ERA, etc.
I think your read of the situation re. mentor ratings is basically correct: increasingly many MATS mentors primarily care about research execution ability (generally ML), not AI safety strategy knowledge. I see this as a feature, not a bug, but I understand why you disagree. I think you are prioritizing a different skillset than most mentors that our mentor selection committee rates highly. Interestingly, most of the technical mentors that you rate highly seem to primarily care about object-level research ability and think that strategy/research taste can be learned on the job!
Note that I think the pendulum might start to swing back towards mentors valuing high-level AI safety strategy knowledge as the Iterator archetype is increasingly replaced/supplemented by AI. The Amplifier archetype seems increasingly in-demand as orgs scale, and we might see a surge in Connectors as AI agents improve to the point that their theoretical ideas are more testable. Also note that we might have different opinions on the optimal ratio of “visionaries” vs. “experimenters” in an emerging research field.
I think that the distribution of mentors we are drawing from is slowing growing to include more highly respected academics and industry professionals by percentage. I think this increases the average quality of our mentor applicant pool, but I understand that this is might be controversial. Note that I still think our most impactful mentors are well-known within the AI safety field and most of the top-50 most impactful researchers in AI safety apply to mentor at MATS.