Stop telling people to pivot to AI safety, I beg you
Epistemic status: Feeling frustrated, and therefore somewhat uncharitable.
I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste’s post.)
There are far more talented people trying to get into AI safety than there are jobs for them.[1] MLAB 2, which I did in 2022, was already competitive to get into, at about 5% acceptance rate (admitted about 40 out of 800 applicants). Things are much, much more competitive now. There is a whole pipeline of AI safety fellowships (and guides for applying to them). It is rumored that one recent cohort of the Anthropic Fellows program had over 10,000 applicants. (Anecdotally, I applied to the Anthropic Fellows program in early 2026, after working at Redwood, METR, and an ML startup, and I didn’t get in!)
And, uh, just take a look at the AI safety jobs? The bottom of the salary range for METR’s current “Member of Technical Staff, Embedded Assessments” job posting is $402k. This is higher than the median total comp for Research Scientists ($310k) and Machine Learning Engineers ($282k). Plus these jobs are high status too. These are really good jobs and nobody needs to be convinced to want them.
I think it’s Actually Bad to keep saying AI safety is talent-constrained. I think it really harms people to tell them they should drop what they’re doing to switch to AI safety when there isn’t a path for the vast majority of the people who do to thrive. These are good, talented people who just want to help, and they could be doing lots of other good things instead of putting themselves through the meatgrinder of applying to AI safety jobs.
Finally, I worry that believing that the field is talent-constrained is holding us back as a field. If we believed that we just needed to wait for the right talent to come along, then we can stop looking elsewhere and fixing fixable problems now. I worry that the “AI safety industrial complex” has in fact gotten too effective at getting new, amazing talent into the field, so that the existing orgs can afford to continue to churn through them without becoming better. If a regular startup treated talent the same way as AI safety orgs, it would fail very quickly and be forgotten. No, you do not keep saying to the world that your field is “talent-constrained” while doing “field building” at top universities. You get to work with your existing team and build the damn thing.
This seems like bad advice that makes the world much worse off in expectation. These orgs offer highish salaries (sometimes) because they’re desperately trying to attract better talent. Talent lies along a continuum. It’s not a binary question whether talented people are or are not trying to get into AI safety and often failing (they are, the majority of applicants for any half-decent job are rejected). It’s about whether all of the important roles are being filled with people who are able to carry them out close to as quickly as possible, or not, because of a dearth of sufficiently strong applicants. I know for a fact that the latter is true.
These organizations are not spending huge amounts of time and energy trying to hire for the fun of rejecting people. They desperately need more people, and they are not finding people with the skills and talents that they need as quickly as they need to get the work done that will make us safer.
Yes, rejections suck, and people should be clear that the talents needed are specific and often at a very high level, and most people won’t be the right fit for most roles. But getting this work done well is much more important than hurt feelings.
Also, your own point cuts against you. If these organizations were actually getting the talent they wanted, they would presumably be saving their money for other things, not offering high salaries.
I agree, but I think they are being too inefficient with their hiring, and bad at utilizing the talent that’s there already, and they are only able to stay this incompetent / inefficient because we have such a great pipeline of new AI safety talent lining up to work for these orgs that they can continue being bad at hiring and working with their existing talent. I think they should get better at these things instead of keeping telling more people to switch to AI safety. (I don’t have actual, recent evidence of this, I worked at safety orgs in 2022-2024, so make what you will of this.)
So basically, you’re telling people not to work on the most important thing in the world that might kill all of us and destroy everything we care about if people don’t get it right on the basis of something that you don’t have actual recent evidence of? Based on experiences that happened between 2022 and 2024? Maybe that should, uh, go near the top of the post.
Also, maybe you should describe those experiences and how they’re evidence for your hypothesis.
Hm, I’m not saying that people should not work on AI safety. I just think we currently overcommunicate this message and it’s bad to do so. I think there is a difference?
The OP is right. At this point, expanding the application pool cannot be that helpful. Talent is better found through head-hunting and more competitive salaries. My guess is that the problem lies more in the fact that a lot of talent is lost to work on capabilities at a frontier lab.
The “AI is an existential risk and you can make a difference” is such a strong attractor for well-meaning people that orgs should really be upfront about what they are looking for. I believe at this point that there is enough misleading information about who should get into AI Safety that the orgs need to start addressing it and cut the “anyone can apply” job postings.
I think it is fair to be clear to young and hopefulls especially, that there is a need for highly competitive talent, not just anyone. I think that communicating to people that this is a highly competitive area both helps keep applicants realistic about their chances, (and help communicate to them whether they should be trying at all) and also serves to bolster the recognition/status of these positions, which makes them more attractive to highly competitive applicants.
Most of the med students I work with are not going into medicine just because it pays well; there are easier ways for these highly talented individuals to make money. No, most of them are highly driven by the status and recognition of the job.
This does lead me to wonder… Are there ways we could work to build recognition and status, as an indirect incentive, for those working in AI safety? It seems silly, but I think the chance of recognition can be disproportionately motivating for many people.
I would expect that if this were true they would offer more paid internships at one tenth the salary of a “Member of Technical Staff, Embedded Assessments” and train some of the many bright people interested in AI safety positions.
I think there’s a major bottleneck of really talented people, and that the field is not great at helping good but not amazing people be useful. These are pretty unfortunate dynamics in combination. But I do a lot of AI safety hiring, and even among people I hire there are big differences in ability which mean big differences in impact. If field building had not gotten those people into AI Safety, the world would be worse off
For each role, the impact different people would have in that role is heavy-tailed. So you can have loads of applicants (e.g. 100-to-1), but doubling your applicant pool would still yield big gains. The maths-y way to say this is that X is a distribution where E[max({X_i : i < 2n})]/E[max({X_i : i < n})] decays slowly in n, like Pareto or Lognormal. I guess this sucks for all (2n-1) people though.
Maybe the problem is partly the fellowships? Like, ideally, people would work on a cool project for a couple weeks (which is a commitment but exactly a blood oath) and if they aren’t great then they can go back to whatever they were doing before, or switch to roles which aren’t so heavy-tailed. But this would require fellowships to be more honest with their participants about their skills, which is unlikely to happen imo.
Whether this is true depends on how effectively you can filter your applicant pool, the marginal cost of processing additional applications, and the quality of the marginal applicants.
Yep.
As someone with a career in technical AI safety, I think communicating about the threats (to politicians and media) is more impactful/important than technical AI safety research.[1]
AI safety is not (and shouldn’t be) just about technical research. I urge people to actually consider comms. People with or without experience can have impact via meeting with their representatives, see How a cold email got the Finnish government to respond on superintelligence regulation.
It seems to me that we definitely need more time and are not on track to solve alignment or prevent extinction through technical means even with more investment. I imagine a lot of AI safety researchers also think we need more time. Please be transparent in your communications.
I agree with this. I’d rather 1000 people doing high-quality research, and 3000 communicating them to policymakers and media. Compared to 1000 people doing high-quality research and 3000 doing low-quality research.
That seems like a false dichotomy. People working on AI safety can also engage in governance and communicate their positions to the media and policymakers, likely garnering even more attention thanks to their technical expertise. In fact, the individuals who have attracted the most media attention recently are precisely those with technical expertise in AI safety. So, the choice isn’t necessarily between working full-time on technical alignment research and working full-time on governance.
There may be low-hanging fruit in making safetyist arguments on social media:
It’s not super difficult to write a comment or reply which gets 1000+ views. Intuitively this is a lot more scaleable than talking to people one-on-one. Due to participation inequality, social media users who speak up can have a disproportionate impact.
If you enjoy social media use anyways, you can do it in your spare time as an entertainment activity.
Social media content gets used to train AIs and used as raw material for RAG answers when people ask chatbots AI safety-related questions.
There are many social media users objecting to AI doom arguments in ways that are easily answered.
VIPs such as congressional staffers presumably read social media sometimes, or even if they mostly don’t, doomer sentiment on social media can still give them cover to take action based on IRL conversations.
I myself have spent a fair amount of time as a “doomer reply guy”. I think I’ve meaningfully shifted sentiment around AI for at least one platform I’ve spent time arguing on. It hasn’t been particularly difficult to become the #1 most active doomer reply guy on my platforms of choice (estimated on the basis that after 100+ hours I haven’t spotted any other plausible contenders).
Fwiw I run a fieldbuilding and research ai safety organization and I agree with this take. We hire aggressively, we don’t do half a dozen performative hiring rounds. Our placement data indicates the majority in our programs end up in some kind of longer-term AI safety job. Nevertheless, I agree with the OP that there should be more honest messaging.
It’s good that there are so many entry opportunities but it would be helpful if there was more clear-eyed messaging about long-term job opportunities. My sense is that compared to eg academia the job market is much better in AI safety but nevertheless many aspiring AI safety researchers/generalists may struggle to find a job for a long time.
I also strongly dislike the weird 2-faced messaging which is at the same time naively idealistic (‘everyone should switch!‘) and elitist (ooooh it’s so hard to find talent). In my not so humble opinion, there is a very self-congratulatory sense of elitism in AI safety insider circles: cf all the talk about ‘high context’. The fact is that AI safety is not a galaxy-brained concept. One can get up to speed with the basics relatively quickly. It is simply not that deep compared to many other fields. That doesn’t mean there can’t be large differences between researchers—there definitely are—but that in many cases this kind of talk comes from often mildly mediocre people who were just early / know the right people / parrot the accepted ideology. In many cases it is more a question of who you know than what you know.
eg as a random guestimate I would guestimate that being up to date with say mid 20th century electrical engineering is intellectually more challenging than being up to date with the sota of interpretability research.
As someone who has very recently pivoted (still pivoting really) to AI safety I pretty strongly disagree with this take.
It seems the real bottleneck right now is organizational capacity to absorb talented people and get them working on important problems in a semi-organized way. Its clear there is so much research to be done and a lot of low hanging fruit to be plucked. We absolutely need more people doing this work. The funding is there and growing fast, expect even more after frontier labs IPO. An announcement like Project Tailwind is a bat signal that the funding is there, the constraint is new orgs to use it effectively. Existing labs can only take on so many new people at a time while staying focused.
But the good news is as that as these organizations and the AI safety ecosystem grows, they will start to be able to absorb more people per year. Like this year we have X number of mentors for AI safety research, they take say 2X mentees. Say 50% of the mentees are really good and become independent researchers, then 6 months from now we will have 2X mentors, so 4X mentees can enter the field.
If you are capable already of being a quite independent researcher, perhaps because you already have a decent amount of research experience in another field, its super valuable for you to pivot because you will not eat up much mentorship time and will still be productive. Even as newcomer with probably not the greatest research taste in this area yet I see the opportunity for so many projects that are interesting, many that I could do mostly independently (writing up my first one now). This would have no drain on organization capacity and so is definitely a net positive.
And I don’t want to be rude, but when it comes to research, there are levels to this shit.
Someone just out of undergrad, even if they are very bright and talented needs a decent amount of handholding to be directed to productive research activities. In my previous field, after a PhD I would say ~33% of people become truly independent researchers who can generate their own good ideas, ~33% have the technical skills but don’t (yet) have the vision to craft their own research agenda, and ~33% never really had the sauce so to speak and needed a ton of hand holding the whole time. (This includes many people coming from top universities, research is just quite different than being a good student). Unsure how this shakes out in the powerful AI age we now in, but I think it may raise the floor but also increase the differentiation in output even more.
Even for senior, established researchers (ie professors), there is a pretty decent gap in productivity. In my previous field of ML+physics probably 50% of the good ideas were coming from the top 5% of the community. Its not that everyone else was doing useless things but they were often not pushing the boundaries in the same way, and more so filling gaps in with incremental work (still valuable! but lower impact).
So getting some exceptionally talented person to switch into AI safety can have huge value. And unfortunately that possibility makes it worth advising as many people as possible to switch into AI safety, to have a chance to get a super talented person doing something impactful. Even if that means some more junior people struggle to find a good home for a while.
Agreed this is more of a bottleneck. And yeah I’m saying that we should not keep telling more people to pivot to AI safety while the ecosystem is unable to absorb them.
If I’m reading you right, it sounds like your main concern is that those who pivot will waste their time? I agree that applying to METR out of the blue won’t get most folks anywhere. And as you say, if you look at the highest-status organizations in the space, the number of openings isn’t nearly enough to satisfy those who will attempt to pivot.
That said, two things:
Provided the frontier labs successfully IPO in the next several months, a lot of their employees will suddenly have liquid wealth. Many of them are safety-motivated, so I expect some will become angel investors, go independent, or fund existing safety orgs. I’d expect the number of orgs hiring for safety-relevant roles to grow noticeably in the next 2 years.
Anecdotally: for most of 2025 I led engineering recruiting at a tiny startup of almost no renown. We received about 3,000 applications a month per position, and about 80% were clearly unqualified. AI is The Current Thing even outside x-risk circles, and the salaries are high, so safety orgs (among others) are going to get a glut of applications no matter what. But I’d guess that most of that glut is also unqualified. Of course, I wasn’t hiring safety roles and the applicants may be more self-selected, but the relevant candidate pool is likely much smaller than the headline numbers. (Still probably large enough to make it a crapshoot for those who are qualified, I imagine.) So hopefully it’s still enough to keep standards high and keep safety orgs staffed with top tier talent
Curious whether your application required any freeform text responses, or just resume? I had similar experience recruiting for ML roles at my last job (non-AI-safety startup)- we received about 1000 resumes for the each position in a few days, but only 50-100 also answered the short round 1 questions that accompanied the application. (And I think the acceptance rate numbers I mention in the post are for “high effort” applications.)
Could it be that technical AI safety is not talent-constrained but that roles other than technical research remain talent-constrained?
My friend’s recent EA forum shortform seems to imply that there is an oversupply of talent on the generalist/operator side also.
I don’t think the linked post implies that? Generalists are extremely hard to hire for, so the fact that those roles are very selective doesn’t mean there’s an oversupply of talent.
Also I don’t think of “generalist” and “operator” as synonyms. I would define them as:
generalist: you can fully trust this person to do basically any task, and to do the meta-task of figuring out what tasks are worth doing
operator: this person is good at executing on operations tasks, which are important and high-skill but do not require strategic thinking—things like filing the paperwork for 501(c)(3) status, setting up your org’s content management system, office management, setting up a hiring process, etc.
My thinking is: of course the standard path opportunity is small—that’s crowded, everyone is being told to take the standard path! Opportunity remains pretty massive if you’re entrepreneurial and somehow summon the slack to Simply Start Doing Thing[^1]. Pretty huge distinction.
[1]: I should have a post about this, soon (TM)
Whoever first said about AI safety that it is “talent-constrained”, whom I don’t know, I wonder if they meant that the people doing AI safety are not good enough to get their task done, and it would be nice to cast a large net and somehow find even more competent people. Like they want the top 1⁄100′000′000 instead of the top 1⁄100′000.
This is true is we see AI safety as a technical project—power laws apply.
What I worry about is how this messages for AI safety as a political project. Sure, safety research is more important than “hurt feelings” but those feelings are a real input into political movements. If you tell someone they’re not “talented” enough to contribute to your movement’s goals and keep repeating that they’re talent constrained, that’s not a great way to coalition build.
Politics is the mind killer but it’s also the only path we have for a pause or slowdown right now. All the technical alignment in the world may not matter if AI safety convinces potential allies that they are untalented.
Maybe they think that, but I think it’s pretty harmful to think that? Those top 1⁄100,000 people can be working on other good things, and having a better time with their lives. To be clear I also don’t believe there are any problems that only the top 1⁄100,000,000 can solve, and I always felt like people who believe it come from a weird psychology. Even in theoretical physics (my academic field before I left) I think people are much more clear-eyed about the external factors that determine whether or not one can do good work beyond someone’s intrinsic competence/abilities/intelligence.
I found this to be a useful counterpoint to accepted wisdom. However, the post does not address what ought to be done instead? Applying for ambitious AI safety fellowships or jobs is likely to help develop skills that are highly valuable and sufficiently transferable to new contexts, particularly as white collar labor is increasingly AI automated. I don’t think 1-2 years spent trying to transition into AI Safety unsuccessfully is time lost. At the least, this leaves you well positioned for advocacy (“I worked on Y projects with X famous company/person, and completed Z fellowship, and I think AI will kill us all”) and transition into other cause areas (or just Some Job).
So probably I would agree with the post more if it was simply a qualification to the call for AI safety work. Rush to AI safety! But maintain sobriety about your chances and keep an eye toward future pivots.
(I would endorse a version of this post warning people not to rush toward e.g. mastering mech interp. or other narrow techniques, which seems to be the default path for software folks)
We, as a civilization, aren’t doing enough AI safety research, so from that perspective there’s simultaneously too little capital and too little labor.
Existing organizations have a hard time slotting in extra people to existing projects. This leads to high pay and high demand “for the right person,” but low hiring rates and low clarity on what the right person is like.
Starting an organization that actually pays people money is also hard. We’re basically running off a charity ecosystem, with limited money and especially limited legibility and middle-management capacity to allocate money well. But I think the money isn’t that limited, the field can slooowly grow as new organizations get toeholds and build legibility.
Research is hard. If a random software engineer tries to do AI safety research, they will try to pick low-hanging fruit and generally not attack the really key problems.
But at the same time, if the people currently trying to build smarter, more reward-hacky AI as fast as possible pivoted to trying to build AI that actually does good things and not bad things, that would be a huge boon.
It’s not like we need money for a new particle collider before people can do alignment work. The actual cost is mostly just paychecks. You can just exercise free will and do the research.
Free will is not a healthy or efficient way to run a research ecosystem either—maybe what would be good is a harsh government regulatory environment that conscripts the organizational apparatus of frontier AI companies to do AI alignment research. I dunno.
It’s talent-constrained, but not in a form that the current institutions can recognize or have short-term incentives to select for, so none of those $402K jobs are going to be available to the actual talent required.
“current institutions” such as MIRI which spends about $8M per year? Or perhaps Survival and Flourishing Fund, which spends about $29M per year on AI, and which you are a speculation grantor for? Have you considered personally making a list of talented people, checking it twice, and sending it to Malo, Jaan, and Andrew Critch? Or does the problem go so deep that you do not “recognize or have short-term incentives to select for” such talent?
I wrote a bit about this from a more pragmatic angle. It’s a useful compliment to your (accurate IMO) vibes and first-principles based take, being somewhat more concrete. People respond well when I share it.
nit: *complement
I strongly agree! We need more smart people doing non-AI things like biology. (My startup Ovelle Bio is currently hiring for in vitro gametogenesis research.) Of course AI safety and AI pause advocacy are definitely needed, but non-AI approaches to human empowerment are overlooked right now.
AI Safety is very important and on the margin it would be better if the field had more talent working on solving the problems
There are not enough senior researchers/engineers to mentor/manage the large number of junior researchers/engineers, meaning most of them are not able to work on solving the problems
The hiring process is brutal. Applications to fellowships or other programs commonly take 8+ hours to complete and do not communicate requirements, leading to a large portion of applicants wasting at least a day each on applications they never had a shot at.
Encouraging people to drop everything and put themselves through the above is a bad idea
The solution to the talent constraint is to train more people, not to get more people interested.
With Project Tailwind there’s now an extra $~1 billion in CG funding for new orgs, and significant amounts from Longview, S&FF and Lightcone etc. There’s also loads of hiring for AI safety roles in the private sector, in the frontier labs (if you’re comfortable with that), startups, governments/AISIs, the EU, media, and legal roles across the world. There’s also lots of room for people to work in activism, grassroots stuff, independent content creation etc.
This could be an extra ~5,000 people meaningfully working full-time in AI safety in 2027.
How does the current pool look for mobilising this 5,000 people?
I don’t really know, but my sense is not great. Talent is clustered in the wrong areas (e.g. I’ve met like 10 people in the space who have a credible understanding of how China works), new people entering the field are, on average, less aligned with AI safety values, and/or less brilliant than people concerned about AI safety in 2018.
How big, how aligned, and how high quality would we want the pool to be here?
To put a number to it, I’d want to build a pool of ~50,000 people across loads of fields, as close to the Pareto frontier of capability and alignment as we can get.
There are lots of ways that the fellowship pipeline should be improved. It’s mostly just too research heavy. I also agree that the field has challenges absorbing these people, especially very capable but not super agentic people.
But most people reading this should strongly consider pivoting to AI safety. This is the defining mission of our time, and if you’re smart, aligned and agentic, you have an excellent chance of being able to contribute.
It seems pretty odd to me to have concrete career advice either way—I think the community really just need to keep communicating “this is an important issue”, and those who have passion in this field would switch, and stay. Talent is one thing, but passion is another important factor IMO.
According to recent studies, there are currently only a few thousand people people working in AI Safety (across labs, academia, donor-funded researchers and orgs, government safety orgs, etc). We have been making significant progress in AI Alignment in the last few years, but is appears likely that time is really short: we need to make a lot more progress a lot faster. So we need more people and enough more funding to fund them. Most people are expecting a lot more funding after the Anthropic IPO (and as a result existing funders are already expending their funds faster) — if so, then shortage of talented people definitely will become the gating factor, if it isn’t already. Also, the labs seem to now feel that AI Safety is becoming a gating factor on what they can ship, so they are likely to start devoting serious money to it (i.e. 9-figure sums).
I think there is also a misconception about (some) fellowships, which you touched on a bit. On paper, a fellowship should help you navigate your career transition to AIS and most will mention that you don’t need previous experience in safety. In practice:
many fellows already did a fellowship before/ worked in AI research/ published papers before/ worked at a frontier lab. I think there‘s been plenty of posts here about that (and if not, there’s definitely been on other channels)
What happens after? I know so many ex-fellows that still didn’t have a full time job/ concrete plan at the end of their ventures