If you have feedback for me, you can fill out the form at https://w-r.me/feedback .
Or you can email me, at [the second letter of the alphabet]@[my username].net
benwr
Karma: 907
Ernyyl ernyyl cbjreshy synfu yvtug. Cbvag vg ng fbzr irel sne njnl zngrevny jubfr genwrpgbel lbh pna cerqvpg. Fbzrobql jnagf gb renfr lbhe qngn? Gbb onq; gur orfg gurl pna qb vf oybpx crbcyr sebz ernqvat vg jura vg’f ersyrpgrq. Nyfb arng cebcregl bs guvf vf gung, vs lbh’er fhssvpvragyl pnershy, vg’f “ernq-bapr”.
Are there any storage media that are basically impossible to destroy/erase? Answer in rot13 ITT.
Note that we made all our code available (https://github.com/PalisadeResearch/shutdown_avoidance/) and it’s pretty easy to run exactly our code but with your prompt, if you’d like to avoid writing your own script. You need a docker runtime and a python environment with the dependencies installed (and it’s easier if you use nix though that’s not required), but if you have those things you just have to modify the
conf.pyfile with your prompt and then do something likerun --model openai/o3.
There are lots of reasons that a “survival drive” is something we were interested in testing; one reason is that self-preservation has been suggested as a “convergent instrumental goal”; see https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf for a (fairly dated, but seminal) discussion of this.
There are lots of ways for systems to malfunction, but only a small number of those involve actively / surprisingly disabling a shutdown mechanism without any instruction to do so. If the issue was that it got the math problems wrong, that seems like a run-of-the-mill malfunction; if it rewrites the shutdown program, that seems like a much more interesting and specific failure mode, and the word “sabotage” seems appropriate.
I weak-downvoted this comment; I did so for the following reasons:
(a) “Reasoning models as trained do not follow a typical instruction hierarchy of a chat template.” doesn’t seem true if you mean that they’re not intended to follow such a hierarchy (see the link to the system card in the OP), and if you mean that they don’t follow it despite being meant/designed to, I don’t really see much evidence presented (and don’t have any background knowledge that would suggest) that the “Goal, Outcome, Warning, Context” sequence is “bound to” give consistent results.
(b) The comment seems pretty unclear about what you tested and why; e.g. the post you link as an explanation seems to be mostly about irrelevant things, and I don’t really want to read the whole thing to figure out what you mean to reference in it. I also don’t understand the setting in which you ran this experiment: Were you using the chat interface? If so, there’s an existing hidden system prompt that’s going to change the results. Were you using our code from github but with your prompt substituted? If so, why only run it 15 times for one model? Your own code with the OpenAI API? Same question, and also I’d like more details in order to understand specifically what you tested.
(c) We explain that a “survival drive” is merely one hypothesis, and mostly not one that we feel is a good explanation for this behavior; far from jumping to anthropomorphic conclusions, we came away feeling that this isn’t a good explanation for what we’re seeing. We don’t mention consciousness at all, as we don’t think it’s very relevant here.
I wouldn’t have downvoted if I hadn’t felt that the comment was mostly based around a strawman / semi-hallucinated impression of the OP, or if it had been clearer about what exactly the takeaway is meant to be, or what exactly you tested. I didn’t strong-downvote because I appreciated that you shared the prompt you ran, and in general I appreciate that you put in some effort to be empirical about it.
Right, I think it just seems like doing emotional preparation that matches this description is a kind of earthbender-friendly / earthbender-assuming move, while an airbender-friendly move would be more like “notice and accept that you’d have more fun doing it a different way or doing a different thing; that flailing isn’t actually fun”. The effect is kind of similar, i.e. both earthbenders and airbenders should come away less-clinging-to-something, but the earthbender comes away less-clinging-to “the locally easy and straightforward things will work if I do them enough” while the airbender is less-clinging-to something more like “This is what I’d choose”.
Re the last-resort framing, I’m not sure why I said that exactly; I think it’s related to the vibe I got from the OP: Like, “if you notice that you’re not making progress, what do you do? Well, you could keep flailing or avoidantly doomscrolling, or you could [do the thing I’m suggesting], or you could give up in despair”; I think it feels like a “last resort” because the other realistic options presented are kind of like different kinds of death?
Yeah I can try to say some of them, though my sense of the crucialness here does shift on learning that you mean for this to be about hour-to-week levels of effort. I guess I may as well try to come up with element-bending-flavored ones since I’m in pretty deep on that metaphor here.
The biggest differences in which one I’d recommend as a “default last resort” depend on the person and their strengths rather than the situation.
The thing recommended for the airbender above, i.e. “retreat to a safe distance and consider if there’s anything that seems like a fun challenge instead of a grind”.
For a firebender: “Commit to doing it really hard for about [30 minutes]. See where you get. Wait [2 * 30 minutes] after that, and check if you feel energized or drained. If energized, repeat; if drained, [I don’t know; this is the one I have the least familiarity with]”
For a waterbender: Something like “Do naturalism to it”. “When you make good enough observations, you can’t help but make high-quality inferences” is one of my favorite quotes, and I think it applies especially well in this kind of last-resort setting.
I think I maybe mean to say a slightly different thing than came across, which makes sense because I was leaning heavily into the metaphor rather than trying to be very clear.
I think the triggers are definitely hints in the direction that buckling up might be the right move. Yet I also observe that, when I imagine the median or even 80th percentile person-explicitly-buckling-up-for-something-big, a big part of me wants to shake my head and be like “ah well, it was nice while it lasted” about their chances of doing a hard thing, especially an unusual hard thing.
This part of me is clearly wrong sometimes: My head would have shaken well off my shoulders if someone had told me-transported-to-1986, “Andrew Wiles is going to spend the next 6 years trying in secret solitude to prove Fermat’s Last Theorem”.
But also I think it’s clearly not entirely mistaken about such a person’s odds. If a person is a native “airbender”, i.e. they are deeply familiar with the “taking their situation lightly” stance, I don’t think I want to recommend that they take a sense of “welp, I guess I have to finally buckle up for this one” at face value, especially in the context of it being a last-resort for one of their most challenging projects or goals. It feels to me like such a person is more likely to succeed by (a) deciding to stop flailing, (b) retreating to a safe distance, and (c) reevaluating whether this is the path they really want to follow, while in connection with their sense of fun.
Sure; if it’s not obvious they’re from the universe of Avatar: The Last Airbender.
Earthbending is substantially about: facing things head-on, “just getting it done”, “buckling down” (though I suppose this can be different than “buckling up”), being unyielding, orienting around “grit”.
Waterbending is substantially about: Being flexible, responsive to the environment, and careful.
Airbending is substantially about: Freedom of movement and action, using an opponent’s strength against them (which in PvE looks more like “doing what’s easy and/or fun”), speed.
The other one is firebending, but I didn’t reference it and I don’t really understand it well enough to put it in the same terms; still, my best gloss attempt is that it’s about focused and kinda bursty / lower-endurance intensity.
In the universe there are (usually) only these four elements, and most benders are pure specialists / only physically capable of learning their particular kind of bending; the main character (the avatar) is the only person who can / has to learn all four. In my analogy these elements don’t really cover the space of motivational structures that well, and anyway people don’t have to be specialists.
You don’t really mention <a thing that I think is extremely crucial> in this domain, which is that you do not have to (metaphorically) be an earthbender about everything. Other types of bending also exist. If you are not a native earthbender, you might be able to learn to do it (the real world does not have only one Chosen One who can bend all the elements), but as a meta-waterbender I personally recommend first looking around carefully, and trying to figure out how the most successful benders of your native element are doing it.
You do seem to see earthbending as maybe a “last resort” rather than the only way to do things, but it’s not obvious to me that it’s the correct last resort for everyone. The last resort of a successful airbender is probably more like “take even more steps back to see if there are any easier but more oblique approaches to this summit, or even other summits you’d actually rather climb”;
I mean to use “the precise definition” to identify it as the one that isn’t “the specific definition” (based on the earlier comparison), not to say that it’s the only precise definition or anything like that. i.e. I could have said “the comparatively more precise definition of these two” instead.
I wonder if we should bet about something here. It seems plausible that we would make different predictions about how much agreement there would be on what is a “chemical”, if you were to explain about the structure, manufacturing, and acquisition of a given substance.
I think we disagree about what a definition is, in natural language.
I think it’s fair to say that you and I have an internally-consistent-though-vague definition of the word “sandwich”, even though I’m sure there are astronomically many edge-case examples for which we would answer differently. And in fact this is probably true for almost literally every non-mathematical noun or verb that we both “know”, albeit to varying degrees.
Edited to add: It suddenly seems likely to me that I should be using the word “meaning” rather than “definition” here, but the rest of the post goes through fine if I make that switch.
I think it seems like a fine possibility in principle, actually; sorry to have given the wrong impression! It’s not my central hope, since strategy-stealing seems like it should make many human-augmentations “available” to AI systems as well. This is notably not true for things involving, e.g., BCIs or superbabies.
When I’m thinking about this, it seems kind of fine if the goalposts move—human strategic capacity will certainly move over time no matter what, right? Like, someone invented crowdfunding and suddenly we could do types of coordination that we previously couldn’t do.
Nate Soares points out that the first paragraph is not quite right: Imagine writing a program that somehow implements an aligned superintelligence, giving it as an objective, “maximize utility according to the person who pressed the ‘go’ button”, and pressing the ‘go’ button.
There’s some sense in which, by virtue of existing in the world, you’re already kind of “lucky” by this metric: It can take a finite amount of information to instantiate an agent that takes unbounded actions on your behalf.
I asked Deep Research to see if there are existing treatments of this basic idea in the literature. It seems most closely related to the concept of “empowerment” in RL, which I’m surprised I hadn’t heard of: https://en.m.wikipedia.org/wiki/Empowerment_(artificial_intelligence)
The Wikipedia article makes it seem like this might also be how RL people think about instrumental convergence?
Human information throughput is allegedly only about 10-50 bits per second. This implies an interesting upper bound, in that the information throughput of biological humanity as a whole can’t be higher than around 50 * 10^10 = 500Gbit/s. I.e., if all distinguishable actions made by humans were perfectly independent, biological humanity as a whole would have at most 500Gbit/s of “steering power”.
I need to think more about the idea of “steering power” (e.g. some obvious rough edges around amplifying your steering power using external information processing / decision systems), but I have some intuition that one might actually be able to come up with a not-totally-useless concept that lets us say something like “humanity can’t stay in ‘meaningful control’ if we have an unaligned artificial agent with more steering power than humanity, expressed in bits/s”.
I think you may have missed, or at least not taken literally, at least one of these things in the post:
The expansion of “superhuman strategic agent” is not “agent that’s better than humans at strategic reasoning”, it’s “agent that is better than the best groups of humans at taking (situated) strategic action”
Strategic action is explicitly context-dependent, e.g. an AI system that’s inside a mathematically perfect simulated world that can have no effect on the rest of the physical world and vice versa, has zero strategic power in this sense. Also e.g. in the FAQ, “Capabilities and controls are relevant to existential risks from agentic AI insofar as they provide or limit situated strategic power.” So, yes, an agent that lives on your laptop is only strategically superhuman if it has the resources to actually take strategic action rivaling the most strategically capable groups of humans.
“increasingly accurately” is meant to point out that we don’t need to understand or limit the capabilities of things that are obviously much strategically worse than us.
I think it probably makes sense for ~everyone to have an explicit list of “things I’d like AI to do for me”, especially around productivity and/or things that could help you with world-saving. If you have a list like this, and we happen to hit a relevant capability threshold before we lose, you should probably avoid wasting time on that thing as quickly as possible.
I made a thing that generates strong passwords (>92 bits) that are also easy to remember because they’re rhyming nonsense couplets (that also scan pretty well): https://www.benwr.net/2025/07/16/opensesame.html