Co-Executive Director at ML Alignment & Theory Scholars Program (2022-present)
Co-Founder & Board Member at London Initiative for Safe AI (2023-present)
Manifund Regrantor (2023-present) | RFPs here
Advisor, Catalyze Impact (2023-present) | ToC here
Advisor, AI Safety ANZ (2024-present)
Ph.D. in Physics at the University of Queensland (2017-2023)
Group organizer at Effective Altruism UQ (2018-2021)
Give me feedback! :)
This is a hard question to answer precisely, as we have changed the metrics by which we have evaluated potential mentors several times. The average quality of mentors we accept has grown each program, by my lights. I weakly think that the average quality of mentors applying has also grown, though much slower.
Update on MATS applications trends:
Mentor applications seem to be growing 2.2x/year and we accepted 20% as primary mentors for 10.0 (Summer 2026).
Fellow applications seem to be growing 1.8x/year (counting only applicants who applied to at least one mentor) and we plan to accept 5% for 10.0.
In contrast to the apparent exponential growth in AI conference attendees, the number of AI publications since 2013 has grown quadratically (data from the Standford HAI 2025 AI Index Report). Quadratic growth in publications suggests that a linearly (constantly) increasing number of researchers are producing publications at a linear (constant) rate. Extrapolating a little, this growth rate suggests there will be 3.7M cumulative publications by 2030 (since 2013).
If the AI researcher growth rate is linear, the exponential growth of AI conference attendees might be due to increased industry presence. Also, it’s possible that the attendee growth rate is also quadratic.
Open question: how fast did the field of cybersecurity grow since the launch of the internet?
Attendees at the top-four AI conferences have been growing at 1.26x/year on average. Data is from Our World in Data. I skipped 2020-2021 for all conferences and 2022 for ICLR, as these conferences were virtual due to the COVID pandemic and had increased virtual attendance.
One could infer from these growth rates that the academic field of AI is growing 1.26x/year. Interestingly, the AI safety field (including technical and governance) seems to be growing at 1.25x/year.
The top-10 most-cited papers that MATS contributed to are (all with at least 290 citations)
Representation Engineering: A Top-Down Approach to AI Transparency
Sparse autoencoders find highly interpretable features in language models
Refusal in language models is mediated by a single direction
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
The Reversal Curse: LLMs trained on” A is B” fail to learn” B is A”
Finding neurons in a haystack: Case studies with sparse probing
Compare this to the top-10 highest-karma LessWrong posts that MATS contributed to (all with over 200 karma):
Here is a plot of the annual citations received by MATS, EleutherAI, and Apart research, adjusted so they start on the same year. The three organizations are somewhat comparable, as they leverage large networks of external collaborators: MATS mentors/fellows, EleutherAI Discord, Apart sprint participants.
The EleutherAI data fits a logistic curve perfectly, asymptoting to ~18.5k citations/year. I can’t fit the others as at least 4 data points are needed to fit a logistic curve.
I made a Google Scholar page for MATS. This was inspired by @Esben Kran’s Google Scholar for Apart Research. Eleuther AI subsequently made one too. I think all AI safety organizations and research programs should consider making Google Scholar pages to better share research and track impact.
Great post! I will fund this project on Manifund.com
Gemini 3 estimates that there are 15-20k core ML academics and 100-150k supporting PhD students and Postdocs worldwide. If the TMLR sample is representative, this indicates that there are:
~20k academics interested in any of the above research areas.
~15k academics interested in the non-robustness research areas.
~5k academics interested in AI safety or alignment (note that this might include RLHF).
I analyzed the research interests of the 454 Action Editors on the Transactions on Machine Learning Research (TMLR) Editorial Board to determine what proportion of ML academics are interested in AI safety (credit to @scasper for the idea).
14% of editors listed any of “robustness”, “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest.
10% of editors listed any of “trustworthy”, “interpretable”, “safety”, “alignment”, “evaluation”, or “security” as a research interest. I excluded “robustness” as much of this research is considered more “capabilities” than “safety”.
3.7% of editors listed “safety” or “alignment” as a research interest.
a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence
Sounds like PIBBSS/PrincInt!
Cheers, Cleo!
Even if AI safety founders should be risk averse, I think we should do better at supporting the relatively few competent founder-types who are deeply interested in AI safety.
I suspect that we disagree significantly on the potential downside risk of most AI safety startups. I think it’s relatively hard to have a significant negative impact, particularly one that outweighs the expected benefits, given how much optimization pressure is being applied to advancing AI capabilities across the economy. Creating a new frontier AI company (e.g., Mistral-sized) or a toxic advocacy org would be notable exception. Maybe Mechanize and Calaveras are exceptions too?
Note that at least Anthropic has a hard time finding talent that is also mission-aligned, which they prefer, particularly for safety teams.
Thanks for sharing, Akshyae! Based on the DMs I received after posting this, I think your experience is unfortunately common. Great job sticking at it and launching Explainable!
Can you share which?
True, but we accepted 75% of all scholars into the 6-month extension last program, so the pressure might not be that large now.
Of note: when I first approached you about becoming a MATS mentor, I don’t think you had significant field-building or mentorship experience and had relatively few papers. Since then, you have become one of the most impactful field-builders, mentors, and researchers in AI safety, by my estimation! This is a bet I would take again.
I think that the distribution of mentors we are drawing from is slowing growing to include more highly respected academics and industry professionals by percentage. I think this increases the average quality of our mentor applicant pool, but I understand that this is might be controversial. Note that I still think our most impactful mentors are well-known within the AI safety field and most of the top-50 most impactful researchers in AI safety apply to mentor at MATS.