Software Engineer interested in AI and AI safety.
Stephen McAleese
Karma: 1,448
Shallow review of technical AI safety, 2025
Epoch AI has a map of frontier AI datacenters: https://epoch.ai/data/data-centers/satellite-explorer
Death and AI successionism or AI doom are similar because they feel difficult to avoid and therefore it’s insightful to analyze how people currently cope with death as a model of how they might later cope with AI takeover or AI successionism.
Regarding death, similar to what you described in the post, I think people often begin with a mindset of confused, uncomfortable dissonance. Then they usually converge on one of a few predictable narratives:
1. Acceptance: “Death is inevitable, so trying to fight it is pointless.” Given the inevitability and unavoidability of death, worrying about it or putting effort into avoiding it is futile and pointless. Just swallow the bitter truth and go on living.
2. Denial: Avoiding the topic or distracting oneself from the implications.
3. Positive reframing: Turning death into something desirable or meaningful. As Eliezer Yudkowsky has pointed out, if you were hit on the head with a baseball bat every week, you’d eventually start saying it built character. Many people rationalize death as “natural” or essential to meaning.
Your post seems mostly about mindset #3: AI successionism framed as good or even noble. I’d expect #2 and #3 to be strong psychological attractors as well, but based on personal experience, #1 seems most likely.
I see all three as cognitive distortions: comforting stories designed to reduce dissonance rather than finding an accurate model of reality.
A more truth-seeking and honest mindset is to acknowledge unpleasant realities (death, AI risk), that these events may be likely but not guaranteed, and then ask what actions increase the probability of positive outcomes and decrease negative ones. This is the kind of mindset that is described in IABIED.
I also think a good heuristic is to be skeptical of narratives that minimize human agency or suppress moral obligations to act (e.g. “it’s inevitable so why try”).
Another one is the imminent prediction that AI progress will soon stop or plateau because of diminishing returns or limitations of the technology. Even a professor I know believed that.
I think that’s a possibility but I think this belief is usually a consequence of wishful thinking and status quo bias rather than carefully examining the current evidence and trajectory of the technology.
In 2022 I wrote an article that is relevant to this question called How Do AI Timelines Affect Existential Risk? Here is the abstract:
Superhuman artificial general intelligence could be created this century and would likely be a significant source of existential risk. Delaying the creation of superintelligent AI (ASI) could decrease total existential risk by increasing the amount of time humanity has to work on the AI alignment problem.
However, since ASI could reduce most risks, delaying the creation of ASI could also increase other existential risks, especially from advanced future technologies such as synthetic biology and molecular nanotechnology.
If AI existential risk is high relative to the sum of other existential risk, delaying the creation of ASI will tend to decrease total existential risk and vice-versa.
Other factors such as war and a hardware overhang could increase AI risk and cognitive enhancement could decrease AI risk. To reduce total existential risk, humanity should take robustly positive actions such as working on existential risk analysis, AI governance and safety, and reducing all sources of existential risk by promoting differential technological development.Artificial Intelligence as a Positive and Negative Factor in Global Risk (Yudkowsky, 2008) is also relevant. Excerpt from the conclusion:
Yet before we can pass out of that stage of adolescence, we must, as adolescents,
confront an adult problem: the challenge of smarter-than-human intelligence. This is
the way out of the high-mortality phase of the life cycle, the way to close the window
of vulnerability; it is also probably the single most dangerous risk we face. Artificial
Intelligence is one road into that challenge; and I think it is the road we will end up
taking. I think that, in the end, it will prove easier to build a 747 from scratch, than to
scale up an existing bird or graft on jet engines.
I think you’re overstating how difficult it is for the government to regulate AI. With the exception of SB 53 in California, the reason not much has happened yet is that there have been barely any attempts by governments to regulate AI. I think all it would take is for some informed government to start taking this issue seriously (in a way that LessWrong people already do).
I think this may be because the US government tends to take a hands off approach and assume the market knows best which is usually true.
I think it will be informative to see how China handles this because they have a track record of heavy-handed government interventions like banning Google, the 2021 tech industry crackdown, extremely strict covid lockdowns and so on.
From some quick research online, the number of private tutoring institutions and the revenue of the private tutoring sector fell by ~80% when the Chinese government banned for-profit tutoring in 2021 despite education having pretty severe arms race dynamics similar to AI.
Also that’s an average statistic and the distribution could be very uneven with some key projects having little AI generated code. For example, 90% of code written by AI could mean that there are nine straightforward web apps where AI is writing 100% of code and then a single algorithms codebase that contains the most valuable algorithms (e.g. tokenization, attention calculations) that is mostly hand written.
Andrej Karpathy recently appeared on the Dwarkesh podcast where he said that although he uses AI heavily for web apps, his new Nano Chat project was written with just AI autocomplete without agents.
That’s a good question. One approach I took is to look at the research agendas and outputs (e.g. Google DeepMinds AI safety research agenda) and estimate the number of FTEs based on those.
I would say that I’m including teams that are working full-time on advancing technical AI safety or interpretability (e.g. the GDM Mechanistic Interpretability Team).
To the best of my knowledge, there are a few teams like that at Google DeepMind and Anthropic though I could be underestimating given that these organizations have been growing rapidly over the past few years.
A weakness of this approach is that there could be large numbers of staff who sometimes work on AI safety and significantly increase the effective number of AI safety FTEs at the organization.
Good observation, thanks for sharing.
One possible reason is that I’ve included more organizations in this updated post and this would raise many estimates.
Another reason is that in the old post, I used a linear model that assumed that an organization started with 1 FTE when founded and linearly increased until the current number (example: an organization has 10 FTEs in 2025 and was founded in 2015. Assume 1 FTE in 2015, 2 FTEs in 2016 … 10 in 2025).
The new model is simpler and just assumes the current number for all years (e.g. 10 in 2015 and 10 in 2025) so it’s estimates for earlier years are higher than the previous model. See my response to Daniel above.
I think it’s hard to pick a reference class for the field of AI safety because the number of FTEs working on comparable fields or projects can vary widely.
Two extremes examples:
- Apollo Program: ~400,000 FTEs
- Law of Universal Gravitation: 1 FTE (Newton)Here are some historical challenges which seem comparable to AI safety since they are technical, focused on a specific challenge, and relatively recent [1]:
Pfizer-BioNTech vaccine (2020): ~2,000 researchers and ~3,000 FTEs for manufacturing and logistics
Human genome project (1990 − 2003): ~3,000 researchers across ~20 major centers
ITER fusion experiment (2006 - present): ~2,000 engineers and scientists, ~5000 FTEs in total
CERN and LHC (1994 - present): ~3000 researchers working onsite, ~15,000 collaborators arouond the world.
I think these projects show that it’s possible to make progress on major technical problems with a few thousand talented and focused people.
- ^
These estimates were produced using ChatGPT with web search.
I’m pretty sure that’s just a mistake. Thanks for spotting it! I’ll remove the duplicated row.
For each organization, I estimated the number of FTEs by looking at the team members page, LinkedIn, and what kinds of outputs have been produced by the organization and who is associated with them. Then the final estimate is an intuitive guess based on this information.
Thanks for your helpful feedback Daniel. I agree that the estimate for 2015 (~50 FTEs) is too high. The reason why is that the simple model assumes that the number of FTEs is constant over time as soon as the organization is founded.
For example, the FTE value associated with Google DeepMind is 30 today and the company was founded in 2010 so the value back then is probably too high.
Perhaps a more realistic model would assume that the organization has 1 FTE when founded and linearly increases. Though this model would be inaccurate for organizations that grow rapidly and then plateau in size after being founded.
AI Safety Field Growth Analysis 2025
Thanks for the post. It covers an important debate: whether mechanistic interpretability is worth pursuing as a path towards safer AI. The post is logical and makes several good points but I find it’s style too formal for LessWrong and it could be rewritten to be more readable.
Thank you for taking the time to comment and for pointing out some errors in the post! Your attention to detail is impressive. I updated the post to reflect your feedback:
I removed the references to S1 and S2 in the IOI description and fixed the typos you mentioned.
I changed “A typical vector of neuron activations such as the residual stream...” to “A typical activation vector such as the residual stream...”
Good luck with the rest of the ARENA curriculum! Let me know if you come across anything else.
While I disagree with a lot of this post, I thought it was interesting and I don’t think it should have negative karma.
I haven’t heard anything about RULER on LessWrong yet:
RULER (Relative Universal LLM-Elicited Rewards) eliminates the need for hand-crafted reward functions by using an LLM-as-judge to automatically score agent trajectories. Simply define your task in the system prompt, and RULER handles the rest—no labeled data, expert feedback, or reward engineering required.
✨ Key Benefits:
2-3x faster development—Skip reward function engineering entirely
General-purpose—Works across any task without modification
Strong performance—Matches or exceeds hand-crafted rewards in 3⁄4 benchmarks
Easy integration—Drop-in replacement for manual reward functions
Apparently it allows LLM agents to learn from experience and significantly improves reliability.
These talks are fascinating. Thanks for sharing.
Nice post. Approval reward seems like it helps explain a lot of human motivation and behavior.
I’m wondering whether approval reward would really be a safe source of motivation in an AGI though. From the post, it apparently comes from two sources in humans:
Internal: There’s an internal approval reward generator that rewards you for doing things that other people would approve of, even if no one is there. “Intrinsically motivated” sounds very robust but I’m concerned that this just means that the reward is coming from an internal module that is possible to game.
External: Someone sees you do something and you get approval.
In each case it seems the person is generating behaviors and they there is an equally strong/robust reward classifier internally or externally so it’s hard to game.
The internal classifier is hard to game because we can’t edit our minds.
And other people are hard to fool. For example, there are fake billionaires but they are usually found out and then get negative approval so it’s not worth it.
But I’m wondering would an AGI with an approval reward modify itself to reward hack or figure out how to fool humans in clever ways (like the RLHF robot arm) to get more approval.
Though maybe implementing an approval reward in an AI gets you most of the alignment you need and it’s robust enough.