Software Engineer interested in AI and AI safety.
Stephen McAleese
Karma: 1,698
Apparently the DeepSeek R1 and Kimi 1.5 papers were published on Arxiv on exactly the same day: 22 January 2025 and both are on the same subject: scaling RL for LLMs.
Sounds like another example of of simultaneous discovery similar to how the light bulb was invented by multiple inventors around the same time.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.
Ironically pre-training data describing misaligned AI (a lot of which has been written by AI safety researchers) increases misalignment in LLMs which the authors of this paper call “self-fulfilling misalignment”: https://arxiv.org/pdf/2601.10160
Great article, thanks for sharing. I’ve been confused by consciousness for a long time and reading this article helped. I found the section on Global Workspace Theory particularly informative.
I also discovered that Kaj Sotala wrote a detailed book review of Consciousness and the Brain in 2019.
I think the fundamental reason why accelerating work a lot is challenging is due to Amdahl’s Law: the maximum speedup is constrained by the percentage of the work that can’t be automated or accelerated. A related idea is the theory of constraints in management.
For example, consider the scenario where 50% of tasks can be fully automated and 50% can’t be automated.
In this scenario maybe the first 50% of automatable work takes 5 hours and the other non-automatable work takes 5 hours.
Using AI you completely automate the first 50% of the work to near zero and now you have 5 hours of work left so the formula for the maximum speedup is 1 / (1 − 0.5) = 2 or 1 / (1 - P) where P is the fraction of work that can be automated.
Good post. The tone is open-minded, empirical, and curious rather than bringing in a lot of assumptions to these experiences.
Strong upvote. A really well-written, persuasive, and timely essay. Thank you for writing it.
Recent events seem like a fork in the road, and the AI safety community needs to make sure it goes down the right (non-violent) path.
Some thoughts on why human-written posts and comments are often more fun and interesting to read than LLM-written ones despite the fact that LLM writing is superior in a lot of ways (e.g. vocabulary size):
Succinctness: Due to typing being effortful, this forces human text to be efficient at conveying meaning per word. This makes it easier to read. LLM text in its current form is often too verbose.
Selectivity: People tend to selectively post content they think is interesting. In contrast, since LLMs do not have a reward function for interestingness, their content is often too bland to read.
Other examples: chemical and biological weapons.
One simple solution that I can think of is to rate limit user posting to something like one LW post per day. Then even if people use LLMs to write posts they would have to be selective about which ones to post which increases quality.
Regarding the point on generalization and inner misalignment using the CoinRun example, I recently found a nice paper called Quantifying generalization in reinforcement learning written by OpenAI in 2018. Quote from the paper:
Results are shown in Figure 2a. We collect each data point by averaging the final agent’s performance across 10,000 episodes, where each episode samples a level from the appropriate set. We can see that substantial overfitting occurs when there are less than 4,000 training levels. Even with 16,000 training levels, overfitting is still noticeable. Agents perform best when trained on an unbounded set of levels,
when a new level is encountered in every episode.The key finding is that overfitting and poor generalization is more likely when the RL agent is trained on an insufficient variety of training data and generalization is much better when the RL agent is trained on lots of levels.
This can be explained using the unidentifiability concept mentioned in the post:
When there is limited training data, many different policies including ones that generalize poorly (like always going to the end of the level) achieve the same reward in the training environment. In this case, all these policies are behaviorally indistinguishable from the intended “go to the coin” strategy.
When there is much more and diverse training data, simpler and poorly generalizing policies begin to score poorly on the training data and are ruled out by the training process and the robust, generalizing policy that internalizes the reward function and is more inner-aligned becomes more identifiable.
To summarize, as the amount and diversity of the training data increases, the resulting trained policy that comes from the training process is more likely to generalize.
What if the actions that benefits billions of people alive today is slowing down AI development?
I think this debate is oversimplifying the situation by focusing on the trade-off between two different factors:
Benefits to people currently alive such as better medicine
Harm to future people via extinction
The situation is more complex and involves:
Benefits to current people
Harms to current people
Benefits to future people
Harms to future people
I think, in general, we should usually take actions that benefit the billions of people alive today, or people who will soon exist, rather than assuming that everyone alive today should get negligible weight in the utilitarian calculus because of highly speculative considerations about what might occur in millions of years.
Even if we focus solely on people currently alive, pausing or regulating AI progress could be net positive: although current people stand to benefit from scientific breakthroughs from AI, fast AI development also creates risks like mass unemployment, disruptions of elections, concentration of power, increasing inequality, and generally making it hard for governments to react to the situation. Also human extinction would negatively affect current people by curtailing their lifespan.
I doubt the race to build AGI is best explained by a desire to build AI that benefits the lives of people alive today or in the future.
Instead, a better explanation seems to be individuals, companies, or researchers wanting the prestige of developing breakthroughs, big tech companies simply trying to make a profit and, an arms race dynamic between AI companies because of a lack of coordination to control the rate of AI progress.
Building ASI to reduce existential risk is only net-positive if the step risk associated with ASI is less than the state risk of the status quo for several decades which is doubtful.
I think what he’s saying is that he and others have been promoting the idea of an impersonal longtermist view a lot over the past few years but that he has moral uncertainty and wants to consider other views. So he wrote a paper about the future of AI using a radically different perspective (the person-affecting view) even though he may not agree with it.
Though as you said if he really favors a more impersonal view then he could have done a better job at communicating that in the paper.
Nick Bostrom has a new paper called Optimal Timing for Superintelligence.
Abstract:
Developing superintelligence is not like playing Russian roulette; it is more like
undergoing risky surgery for a condition that will otherwise prove fatal. We
examine optimal timing from a person-affecting stance (and set aside simulation
hypotheses and other arcane considerations). Models incorporating safety
progress, temporal discounting, quality-of-life differentials, and concave QALY
utilities suggest that even high catastrophe probabilities are often worth
accepting. Prioritarian weighting further shortens timelines. For many parameter
settings, the optimal strategy would involve moving quickly to AGI capability, then
pausing briefly before full deployment: swift to harbor, slow to berth. But poorly
implemented pauses could do more harm than good. -- Optimal Timing for SuperintelligenceSome friends and I were wondering why he seems to be emphasizing a person-affecting argument today (focused on the benefits to people alive) whereas previously he emphasized a more impersonal argument (focused on future people not alive today). For example the last sentence in Superintelligence (2014) is this:
In this book, we have attempted to discern a little more feature in what is otherwise still a relatively amorphous and negatively defined vision—one that presents as our principal moral priority (at least from an impersonal and secular perspective) the reduction of existential risk and the attainment of a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment. -- Superintelligence
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
Thanks for the detailed comment.
Outside of toy situations, it’s rare in modern ML training for the solution with the lowest loss on the training data to actually be underdetermined in a meaningful sense.
Would you mind elaborating on what you mean by this? My guess is that you’re saying that in a real-world situation with sufficiently diverse data and enough training, catastrophic misgeneralization or overfitting like the CoinRun example in the post is rare.
Though there is also the problem of adversarial examples (e.g. jailbreaks) where the model performs okay on normal test distribution data but poorly on OOD examples that are designed to trick the model.
After reading The Adolescence of Technology by Dario Amodei I now think that Amodei is one the most realistic and calibrated AI leaders on the future of AI.
I think the level of disagreement among the experts implies that there is quite a lot of uncertainty so the key question is how to steer the future toward better outcomes while reasoning and acting under substantial uncertainty.
The framing I currently like best is from Chris Olah’s thread on probability mass over difficulty levels.
The idea is that you have initial uncertainty and a distribution that assigns probability mass to different levels of alignment difficulty.
The goal is to develop new alignment techniques that “eat marginal probability” where over time the most effective alignment and safety techniques can handle the optimistic easy cases, and then the medium and hard cases and so on. I also think the right approach is to think in terms of which actions would have positive expected value and be beneficial across a range of different possible scenarios.
Meanwhile the goal should be to acquire new evidence that would help reduce uncertainty and concentrate probability mass on specific possibilities. I think the best way to do this is to use the scientific method to proposed hypotheses and then test them experimentally.
RL is much less efficient per token: one scalar reward every several thousand tokens instead of a label per token. But the per-token value is different to the marginal value of each unit of compute invested which is what determines compute allocation. The 80⁄20 split tells you the two are roughly equal in value at the margin.
What makes them equal at the 80⁄20 equilibrium is that pretraining has diminishing returns. For pretraining the relationship between loss and compute is a power law (roughly C^-0.05, so 10x compute reduces the remaining reducible loss by ~11%) and given the size of pretraining runs now, the marginal value of pretraining is low enough to justify allocating 20% of compute to RL.
Also it depends on how you measure efficiency. Pretraining wins on efficiency per token but RL is often better at efficiency per unit of human effort. For example, writing a verifier function for a math problem takes much less human effort than writing out all the steps in the solution.