From what I have seen, I think many folks are looking into that already (that was the big takeaway from this incident). I haven’t seen anyone talking about self improvement though … likely that’s because someone checked and didn’t find anything, but it’d be good to make sure someone got around to checking …
Ephraiem Sarabamoun
Karma: 42
Hey! Thank you for your comment. It’s an interesting way to frame things. I think humans have many different, competing drives and I don’t think that tool use alone explains how we came to dominate the planet. We are good at developing and using tools, but we are also good (at least compared to other animals) at reasoning, anticipating future problems, and delaying gratification to address them. I do not think it was predestined for us to end up where we are today. I don’t think it’s guaranteed that humanity will move forward with AI development especially if the risks become increasingly apparent and society decides that continuing down this path is not worth the gamble.
Gratitude and the End of the World
Did Someone Check if Rogue Agents are Interested in Self-Improvement?
Thank you very much for sharing! In a sense, all capabilities really originate in pre-training. Mathematical ability, research ability, fiction writing ability, etc. all have to come from somewhere in the pre-training distribution. Clearly though, modern models excel in certain subjects such as mathematical ability more than others like fiction writing. I disagree with you in that I do think the fundamental reason for this is post-training. I suppose the key experiment that we can run to see which of us is right is by testing a base-model on mathematics vs fiction writing, and see how performance changes after post-training. We can try this with an open-sourced model.
I would like to also make two points:
1. Most RL-for-math is RLAIF vs RLVR: haven’t encountered this claim before, and I find it surprising. From what I understand, mathematical proofs can often be translated into Lean and mechanically verified. Even if we accept that much of math RL is RLAIF, though, RLAIF could still be substantially more effective for mathematics than for subjective domains. A reward model’s usefulness ultimately depends on the quality and reliability of its supervision. Paying mathematicians to judge whether a proof is correct seems likely to produce a much cleaner signal than paying a typical human evaluator to judge whether a piece of writing is “good.” So even under RLAIF, mathematics benefits from the fact that correctness is much more objectively verifiable than writing quality.
2. You did not say this, but I would caution against making the assumption that because the amount of compute expended on math RL is less that therefore RL plays less of a role in the model’s mathematical abilities. A typical mathematical RL environment may simply be much cheaper to run than a task-completion or coding environment (which likely require much more scaffolding, tool usage, etc.). So comparing raw RL compute across domains may not be 1:1.
Thanks again for sharing, and I do think that your “how much BS is in the pre-training literature” is valuable insight.
This is very interesting! Thank you for sharing. It seems to me that monitorability in some of your experiments rapidly approaches some non-zero lower bound. I was wondering, do you happen to know what factors determine that lower bound? Could it have something to do with the recurrent architecture that you are using (or perhaps with your training setup)?
Thank you for writing this. I am open to, but not yet convinced by, the argument that it is best to keep AI safety nonpartisan. My main contention, though, is that even with every reasonable precaution, I am skeptical that the issue can remain bipartisan for long (I would genuinely be interested to hear if you think I am getting something wrong here). Ultimately, I see this issue becoming partisan for two reasons that I have experienced firsthand:
1. When you talk to political offices about AI Safety and you actually get them to care about the issue, they always seem to convert the issue into a political wedge to attack their political opponents. I.e. politicians are extremely liable to convert any issue from their constituents into a political issue that they can use in their political contest to get elected. For example, I recently heard that some Democratic Party leaders are already discussing the possibility of making AI safety and AI regulation a major political wedge issue in the 2028 election.
2. Even when you talk to regular people about AI Safety, if they get it, they tend to integrate AI Safety into their pre-existing worldview which tends to be partisan. When those individuals then share their views on AI Safety (in person or on social media), they tend to bring along all the partisan trappings of their wider worldview to the message.
I suppose the root reason I am skeptical that AI Safety can remain non-partisan is because, in a sense, once you share the AI Safety message, you can’t really control how the person you are talking to processes and integrates that information into their worldview, and our society is set up in such a way as to naturally pull important issues into existing partisan identities.
You’re completely right, but I’m tempted to fall back on the old aphorism: beggars can’t be choosers. Other than METR, are there any organizations you think could currently fulfill the auditor role?
Thank you very much for your reply. For your first point:
Yes, I already said that (“the achieving-goals part could come into conflict with the map-accuracy part (because deception is often useful for achieving goals)”).
I’d slightly refine this and say (which I think you do say in the original comment), that the contradiction between those two pillars of rationalism come in when you are forming a community. There is no contradiction in an individual seeking to achieve the best map that reflects the territory, and that individual choosing to use deception as a tool to best achieve your goals given your map of the territory. The contradiction comes when you are organizing a community and have to decide whether to orient it towards finding truth “achieving the best map that reflects the territory” or towards pushing a message/political position “achieving your goals given your map”.
For your second point:I’m not interested in being a “moral person” (whatever that means); I’m interested in achieving the map that reflects the territory.
I am new to the ways of rationalism so forgive me if this is incorrect, but I think the best way to map what I mean to your worldview is to say that morality has to do with what you define your goals to be not necessarily with the process of “achieving the map that reflects the territory.” Though if “achieving the map that reflects the territory” is one of the goals that you are trying to achieve, then in my lingo I’d call that a moral position. Or translating completely to my language: “You think pursuing truth is good”.
Thanks again for engaging with me!
Well said, but I wonder, would upholding that level of truthfulness and honesty make one less effective politically? Granted this level of intellectual honesty is admirable, and likely is responsible for the incredible borderline prophetic discussions/attention given to AI Safety in the early rationalist community, but do you think you have to in compromise this standard to be politically effective?
The way I see it, and I wonder what you think of this, is that you could be significantly more deceptive than what you describe and still be a moral person.
I would break it down like this:
Lying for immoral reasons—Morally wrong in all cases
Lying for moral reasons—Highly dubious but can be moral in rare cases (Odysseus and sister simplice in les mis)
Using speech to convince/persuade rather than pursue truth (Sophistry: the domain of lawyers and politicians) - Not immoral unless the ends are immoral.
Speaking the absolute truth to the best of your ability (Socrates: the scientist’s or rationalist’s standard) - Noble, ie a form of moral excellence
Let our scholars think, let our warriors fight?
Thank you for sharing! I’m curious what you see as the advantage of having a well-funded independent group pursue this work rather than building automated alignment-research teams within frontier companies such as Anthropic or OpenAI.
Internal teams would have access to the most capable internal models, substantial funding and compute, and an established institution with considerable technical expertise.
Put more simply, what could a third-party automated alignment-research organization offer that an internal team at Anthropic or OpenAI could not?
I find it more useful to try tracking back such behaviors to training. I suspect that the model learns altruistic/self sacrificing personas in pretraining, and those personas are likely selected for in post-training because
Being helpful, honest, etc (all the things that go into making a well behaved assistant) likely corrolates with altruism/self sacrifice
There is likely RL training that pushes agent coordination to solve long horizon tasks (that likely strongly selects for “pro-social” traits and personas.
Yep, apologies, I meant specifically humanoids or, more broadly, “general-purpose” robots. I take your point that, at least in China, we seem already to have reached an inflection point. My intuition is that robotics today may be roughly where LLMs were in 2023. Not yet broadly economically transformative, but experiencing a rapid surge in investment and technical progress, but with China and the United States effectively swapping places in terms of who is leading the buildout.
Thank you for sharing! I got a kick out of the term “prosaic RSI” haha. I think all your timelines seem reasonable to me as an upper bound. The main place I’d differ is that I expect robot mass production to begin before AGI makes robots economically profitable. As we’ve seen with the compute buildout, investors and governments are perfectly willing to pour money and resources into an industry if they believe it will be strategically critical in the future. So I’d probably move that milestone back by a few years.
The Control Paradox
I would be interested in a full serving of “holy shit” please.
Should AI Safety Researchers Experiment with Automated Research
I don’t think the idea of boltzmann brains and the ideas of a thermalized universe are related. The argument that you offer does effectively refute the boltzman brain problem but does not argue against that our universe is moving towards thermal equilibrium.
This whole idea of “choas” hides a lot of baggage. For every physical system you must define the states of maximal entropy. Those states are only relevant given your system constraints. When people talk of “choas” or maximal entropy I always think that they are in the back of their mind thinking of a homogeneous gas of particles bouncing around. That is the state of maximal entropy for an ideal gas but that does not transfer to other physical systems.
Bringing this back to the subject at hand, the reason that I and other physicists think the universe is handed towards a total thermalization is because of dark energy which is a physical and empirical effect that has been deduced from observations done by independent teams of astronomers. There is still an area where little is understood, and it is an ongoing area of study, but what would demonstrate that this is wrong are empirical results.
One point of clarification that might be useful, the “thermal equilibrium” that current cosmology suggests we are moving towards, bears no relation at all to the thermal equilibrium of an ideal gas where you have a well mixed box of particles bouncing around and occasionally fluctuating into complex objects. If dark energy dominates, then particles and structures will become increasingly causally isolated and all ordinary interactions will eventually die out (so it is the kind of thermal equilibrium that can never give rise to a conscious experience). The thermalized universe current theories of cosmology predict would not lead to the Boltzmann brain problem.
Interesting read, thank you for sharing. I think I would distinguish between two “optimization processes” that have molded the fates of living things on this planet (I don’t love the term “optimization process” but it will do).
1. Evolution by natural selection which selects for “fitness”
2. Human intentional selection, which selects for usefulness to humans.As a general principle, I do think you are on to something with the idea of a “honeymoon phase”. The general pattern is, new niche opens up, one specie moves into this niche and dominates it (honeymoon phase), other species move into the niche making it much less comfortable and much more competitive than it used to be.
I do think though, that with the emergence of humans on the planet, the second optimization process is dominant. Like you mentioned with the example of the chickens, I do think it does make sense to translate the idea of a honeymoon phase to this optimization process as well, but we must account for the fact that unlike natural selection, what is driving the train now are human preferences. This makes me more optimistic than you are that barring major cataclysms or the emergence of superhuman AIs, humanity will have a bright future.
One last note, if AI development continues and we do enter a post human AI age, then the world might be molded by what the AIs want rather than what we want. In that world, humans might indeed enter a honeymoon phase where the AIs treat us well at first and then really poorly after (like the chicken).
Would be interested in hearing your thoughts on this, and thanks again for sharing!
Hey Eliezer, I‘d be very interested to hear how your views have changed in light of how things have played out. Not to fault you for one moment, it’s incredible that you were writing so clearly on this topic 19 years ago! It seems though, that Nature smiled upon those who side stepped the problem. I wonder if you can trace the source of the error in your prediction and how your views have updated since.