Software Engineer interested in AI and AI safety.
Stephen McAleese
Karma: 1,747
Does progress in AI safety require progress in AI capabilities?
Gwern’s Why Tool AIs Want to be Agent AIs blog post has aged well and made accurate predictions about the trajectory of AI.
RL is much less efficient per token but the per-token value is different to the marginal value of each unit of compute invested which is what determines compute allocation. The 80⁄20 split tells you the two are roughly equal in value at the margin.
What makes them equal at the 80⁄20 equilibrium is that pretraining has diminishing returns. For pretraining the relationship between loss and compute is a power law (roughly C^-0.05, so 10x compute reduces the remaining reducible loss by ~11%) and given the size of pretraining runs now, the marginal value of pretraining is low enough to justify allocating 20% of compute to RL.
Also it depends on how you measure efficiency. Pretraining wins on efficiency per token but RL is often better at efficiency per unit of human effort. For example, writing a verifier function for a math problem takes much less human effort than writing out all the steps in the solution.
Apparently the DeepSeek R1 and Kimi 1.5 papers were published on Arxiv on exactly the same day: 22 January 2025 and both are on the same subject: scaling RL for LLMs.
Sounds like another example of of simultaneous discovery similar to how the light bulb was invented by multiple inventors around the same time.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.
Ironically pre-training data describing misaligned AI (a lot of which has been written by AI safety researchers) increases misalignment in LLMs which the authors of this paper call “self-fulfilling misalignment”: https://arxiv.org/pdf/2601.10160
Great article, thanks for sharing. I’ve been confused by consciousness for a long time and reading this article helped. I found the section on Global Workspace Theory particularly informative.
I also discovered that Kaj Sotala wrote a detailed book review of Consciousness and the Brain in 2019.
I think the fundamental reason why accelerating work a lot is challenging is due to Amdahl’s Law: the maximum speedup is constrained by the percentage of the work that can’t be automated or accelerated. A related idea is the theory of constraints in management.
For example, consider the scenario where 50% of tasks can be fully automated and 50% can’t be automated.
In this scenario maybe the first 50% of automatable work takes 5 hours and the other non-automatable work takes 5 hours.
Using AI you completely automate the first 50% of the work to near zero and now you have 5 hours of work left so the formula for the maximum speedup is 1 / (1 − 0.5) = 2 or 1 / (1 - P) where P is the fraction of work that can be automated.
Good post. The tone is open-minded, empirical, and curious rather than bringing in a lot of assumptions to these experiences.
Strong upvote. A really well-written, persuasive, and timely essay. Thank you for writing it.
Recent events seem like a fork in the road, and the AI safety community needs to make sure it goes down the right (non-violent) path.
Some thoughts on why human-written posts and comments are often more fun and interesting to read than LLM-written ones despite the fact that LLM writing is superior in a lot of ways (e.g. vocabulary size):
Succinctness: Due to typing being effortful, this forces human text to be efficient at conveying meaning per word. This makes it easier to read. LLM text in its current form is often too verbose.
Selectivity: People tend to selectively post content they think is interesting. In contrast, since LLMs do not have a reward function for interestingness, their content is often too bland to read.
Other examples: chemical and biological weapons.
One simple solution that I can think of is to rate limit user posting to something like one LW post per day. Then even if people use LLMs to write posts they would have to be selective about which ones to post which increases quality.
Regarding the point on generalization and inner misalignment using the CoinRun example, I recently found a nice paper called Quantifying generalization in reinforcement learning written by OpenAI in 2018. Quote from the paper:
Results are shown in Figure 2a. We collect each data point by averaging the final agent’s performance across 10,000 episodes, where each episode samples a level from the appropriate set. We can see that substantial overfitting occurs when there are less than 4,000 training levels. Even with 16,000 training levels, overfitting is still noticeable. Agents perform best when trained on an unbounded set of levels,
when a new level is encountered in every episode.The key finding is that overfitting and poor generalization is more likely when the RL agent is trained on an insufficient variety of training data and generalization is much better when the RL agent is trained on lots of levels.
This can be explained using the unidentifiability concept mentioned in the post:
When there is limited training data, many different policies including ones that generalize poorly (like always going to the end of the level) achieve the same reward in the training environment. In this case, all these policies are behaviorally indistinguishable from the intended “go to the coin” strategy.
When there is much more and diverse training data, simpler and poorly generalizing policies begin to score poorly on the training data and are ruled out by the training process and the robust, generalizing policy that internalizes the reward function and is more inner-aligned becomes more identifiable.
To summarize, as the amount and diversity of the training data increases, the resulting trained policy that comes from the training process is more likely to generalize.
What if the actions that benefits billions of people alive today is slowing down AI development?
I think this debate is oversimplifying the situation by focusing on the trade-off between two different factors:
Benefits to people currently alive such as better medicine
Harm to future people via extinction
The situation is more complex and involves:
Benefits to current people
Harms to current people
Benefits to future people
Harms to future people
I think, in general, we should usually take actions that benefit the billions of people alive today, or people who will soon exist, rather than assuming that everyone alive today should get negligible weight in the utilitarian calculus because of highly speculative considerations about what might occur in millions of years.
Even if we focus solely on people currently alive, pausing or regulating AI progress could be net positive: although current people stand to benefit from scientific breakthroughs from AI, fast AI development also creates risks like mass unemployment, disruptions of elections, concentration of power, increasing inequality, and generally making it hard for governments to react to the situation. Also human extinction would negatively affect current people by curtailing their lifespan.
I doubt the race to build AGI is best explained by a desire to build AI that benefits the lives of people alive today or in the future.
Instead, a better explanation seems to be individuals, companies, or researchers wanting the prestige of developing breakthroughs, big tech companies simply trying to make a profit and, an arms race dynamic between AI companies because of a lack of coordination to control the rate of AI progress.
Building ASI to reduce existential risk is only net-positive if the step risk associated with ASI is less than the state risk of the status quo for several decades which is doubtful.
I think what he’s saying is that he and others have been promoting the idea of an impersonal longtermist view a lot over the past few years but that he has moral uncertainty and wants to consider other views. So he wrote a paper about the future of AI using a radically different perspective (the person-affecting view) even though he may not agree with it.
Though as you said if he really favors a more impersonal view then he could have done a better job at communicating that in the paper.
Nick Bostrom has a new paper called Optimal Timing for Superintelligence.
Abstract:
Developing superintelligence is not like playing Russian roulette; it is more like
undergoing risky surgery for a condition that will otherwise prove fatal. We
examine optimal timing from a person-affecting stance (and set aside simulation
hypotheses and other arcane considerations). Models incorporating safety
progress, temporal discounting, quality-of-life differentials, and concave QALY
utilities suggest that even high catastrophe probabilities are often worth
accepting. Prioritarian weighting further shortens timelines. For many parameter
settings, the optimal strategy would involve moving quickly to AGI capability, then
pausing briefly before full deployment: swift to harbor, slow to berth. But poorly
implemented pauses could do more harm than good. -- Optimal Timing for SuperintelligenceSome friends and I were wondering why he seems to be emphasizing a person-affecting argument today (focused on the benefits to people alive) whereas previously he emphasized a more impersonal argument (focused on future people not alive today). For example the last sentence in Superintelligence (2014) is this:
In this book, we have attempted to discern a little more feature in what is otherwise still a relatively amorphous and negatively defined vision—one that presents as our principal moral priority (at least from an impersonal and secular perspective) the reduction of existential risk and the attainment of a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment. -- Superintelligence
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
In less than 24 hours, that tweet has reached over 100 million views and 600k likes!
For comparison, a typical Elon Musk tweet (the most followed person on X) seems to get about 10 million views and 10k likes.
The tweet also stands out for being unusually well-written and completely unambiguous about its message.