Co-Executive Director at ML Alignment & Theory Scholars Program (2022-present)
Co-Founder & Board Member at London Initiative for Safe AI (2023-present)
Manifund Regrantor (2023-present) | RFPs here
Advisor, Catalyze Impact (2023-present) | ToC here
Advisor, AI Safety ANZ (2024-present)
Ph.D. in Physics at the University of Queensland (2017-2023)
Group organizer at Effective Altruism UQ (2018-2021)
Give me feedback! :)
SFF-2025 S-Process funding results have been announced! This year, the S-process gave away ~$34M to 88 organizations. The mean (median) grant was $390k ($241k).
Also of interest: MATS mentor applications have been growing linearly at a rate of +70/year. We are now publicly advertising for mentor applications for the first time (previously, we sent out mass emails and advertised on Slack workspaces), which might explain the sub-exponential growth. Giving the exponential growth in applicants and linear growth in mentors, the latter seems to be the limiting constraint still.
Josh Landes shared that BlueDot Impact’s application rate has been increasing by 4.7x/year.
I’ve updated the OP to reflect my intention better. Also, thanks for reminding me re. advertising on LW; MATS actually hasn’t made an ad post yet and this seems like a big oversight!
Any alternative framing/text you recommend? I think it’s a pretty useful statistic for AI safety field-building.
I’m interested in determining the growth rate of AI safety awareness to aid field-building strategy. Applications to MATS have been increasing by 70% per year. Do any other field-builders have statistics they can share?
Here are the AI capabilities organizations where MATS alumni are working (1 at each except for Anthropic and GDM, where there are 2 each):
Anthropic
Barcelona Supercomputing Cluster
Conduit Intelligence
Decart
EliseAI
Fractional AI
General Agents
Google DeepMind
iGent AI
Imbue
Integuide
Kayrros
Mecha Health
Mistral AI
MultiOn
Norm AI
NVIDIA
Palantir
Phonic
RunRL
Salesforce
Sandbar
Secondmind
Yantran
Alumni also work at these organizations, which might be classified as capabilities or safety-adjacent:
Freestyle Research
Leap Labs
Oh, shoot, my mistake.
Sorry, I should have said “~6 on capabilities at a frontier AI company”.
Hi Nicholas! You are not in the data as you were not a MATS scholar, to my knowledge. Were you a participant in one of the MATS training programs instead? Or did I make a mistake?
80% of MATS alumni who completed the program before 2025 are still working on AI safety today, based on a survey of all available alumni LinkedIns or personal websites (242/292 ~ 83%). 10% are working on AI capabilities, but only ~6 at a frontier AI company (2 at Anthropic, 2 at Google DeepMind, 1 at Mistral AI, 1 extrapolated). 2% are still studying, but not in a research degree focused on AI safety. The last 8% are doing miscellaneous things, including non-AI safety/capabilities software engineering, teaching, data science, consulting, and quantitative trading.
Of the 193+ MATS alumni working on AI safety (extrapolated: 234):
34% are working at a non-profit org (Apollo, Redwood, MATS, EleutherAI, FAR.AI, MIRI, ARC, Timaeus, LawZero, RAND, METR, etc.);
27% are working at a for-profit org (Anthropic, Google DeepMind, OpenAI, Goodfire, Meta, etc.);
18% are working as independent researchers, probably with grant funding from Open Philanthropy, LTFF, etc.;
15% are working as academic researchers, including PhDs/Postdocs at Oxford, Cambridge, MIT, ETH Zurich, UC Berkeley, etc.;
6% are working in government agencies, including in the US, UK, EU, and Singapore.
10% of MATS alumni co-founded an active AI safety start-up or team during or after the program, including Apollo Research, Timaeus, Simplex, ARENA, etc.
Errata: I mistakenly included UK AISI in the “non-profit AI safety organization” category instead of “government agency”. I also mistakenly said that the ~6 alumni working on AI capabilities at frontier AI companies were all working on pre-training.
Makes you wonder if there’s some 4D chess going on here. Occam’s razor suggests otherwise, though. And if true, this seems wholly irresponsible, given that AI risk skeptics can just point to this situation as an example that “even if we do no safety testing/guardrails, it’s not that bad! It just offends a few people.” It seems hard to say which direction this will impact SB 53, for example.
I’m open to this argument, but I’m not sure it’s true under the Trump administration.
Technical AI alignment/control is still impactful; don’t go all-in on AI gov!
Liability incentivizes safeguards, even absent regulation;
Cheaper, more effective safeguards make it easier for labs to meet safety standards;
Concrete, achievable safeguards give regulation teeth.
Not sure this is interesting to anyone, but I compiled Zillow’s data on 2021-2025 Berkeley average rent prices recently, to help with rent negotiation. I did not adjust for inflation; these are the raw averages at each time.
I definitely think that people should not look at my estimates and say “here is a good 95% confidence interval upper bound of the number of employees in the AI safety ecosystem.” I think people should look at my estimates and say “here is a good 95% confidence interval lower bound of the number of employees in the AI safety ecosystem,” because you can just add up the names. I.e., even if there might be 10x the number of employees as I estimated, I’m at least 95% confident that there are more than my estimate obtained by just counting names (obviously excluding the 10% fudge factor).
So, conduct a sensitivity analysis on the definite integral with respect to choices of integration bounds? I’m not sure this level of analysis is merited given the incomplete data and unreliable estimation methodology for the number of independent researchers. Like, I’m not even confident that the underlying distribution is a power law (instead of, say, a composite of power law and lognormal distributions, or a truncated power law), and the value of seems very sensitive to data in the vicinity, so I wouldn’t want to rely on this estimate except as a very crude first pass. I would support an investigation into the number of independent researchers in the ecosystem, which I would find useful.
By “upper bound”, I meant “upper bound on the definite integral ”. I.e., for the kind of hacky thing I’m doing here, the integral is very sensitive to the choice of bounds . For example, the integral does not converge for . I think all my data here should be treated as incomplete and all my calculations crude estimates at best.
I edited the original comment to say ” might be a bad upper bound” for clarity.
It’s also worth noting that almost all of these roles are management, ML research, or software engineering; there are very few operations, communications, non-ML research, etc. roles listed, implying that these roles are paid significantly less.
1 vote
Overall karma indicates overall quality.
0 votes
Agreement karma indicates agreement, separate from overall quality.
I can help with this, if you like