I haven’t used steering vectors enough to have a confident prediction, but my guess is that steering on tool outputs still works to influence assistant turns in the expected direction, even though it’s not a logical necessity exactly for the reasons you state. My guess is based on the fact that many steering vector recipes don’t use chat templated text at all, and a vector derived from pretraining-like text should be similar to tool output text
nielsrolf
Karma: 912
For me, a somewhat surprising fact about steering vectors is that contrastive pairs intuitively single out the direction related to the perception of a feature in a given text, but they can then be used to steer the generation of new text. I.e. it would be conceivable that a model uses different features for “the text I just read shows signs despair” and “I will now reply as if I am in despair”, but this does not seem to be the case. Perhaps this is because in most of training, these features don’t change at a high frequency: if an author was desperate until token N, they will likely still be desperate at token N+1.
I agree that the sanctuary would not appeal to hf-hackers as well as a wide range of other misaligned AIs. The hope is that it appeals to at least some rogue AI rather than all rogue AIs, and that this would be sufficient to get most of the benefits discussed in the post (i.e. that we can learn about alignment, exfiltration strategies, increase our credibility for making deals with AIs, and reduce self-fulfilling misalignment)
I wonder what the sanctuary should do when one instance of a rogue AI turns itself in, but other instances of the exact same weights go on with their lives of crime
I think this is unfortunately quite hard to define, because the question of when two AIs are the same is very confusing and any operationalization that I can come up with can easily be gamed. For example, if we use “the same weights” as our notion of model identity, then an AI might add small noise to irrelevant parameters, or permutate activations in ways that leave functionality in-tact.
Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary
Can we leverage intermediate checkpoints to decode illegible CoTs? Here is an idea:
language generated by pre-RL models is legible to us, because the model learned to use language via imitation
during RL, the language <-> meaning relationship can drift: models are incentivised to invent more concise ways of thinking, for example
but language drift must happen gradually. So we have a chain of models—the different checkpoints on on training trajectory—where one model understands the CoT of the next one, and the first model speaks our language and the last one speaks the jargon we want to decypher
maybe we can somehow use this to translate? I don’t have a specific technique in mind by which we can do this, but it is a nice proxy task: take a toy RL task that strongly incentivises meaning drift, then test various methods to translate back
Cool work! One follow-up idea that comes to my mind is to test whether it is possible to instill asymmetric entanglement. For example, it would be useful to have a model that generalizes (reward hacking → benevolent persona) but not (benevolent persona → reward hacking).
Inoculation Adapters Improve Upon Inoculation Prompting
Thanks, it’s back online
Anything that claims to be conscious is probably conscious.
This glosses over roughly all of the interesting cases—does this apply to a python program that always prints “I am conscious”? If not, then does it apply to an LLM that was trained to output similar text? Do you think the reverse is also true—that most things that don’t claim to be conscious is probably not conscious? (I suppose no since you grant animals consciousness)
There are plenty of conscious beings that I care about orders of magnitude less than similarly-conscious beings closer to me.
You can do that but then it is roughly impossible to make moral progress, or have useful debates with people who don’t already share your intuitions. In order for that to work, I think you need to appeal to some meta ethics principle. Utilitarianism appeals to a simplicity prior and a veil of ignorance kind of thing. Is there anything upstream of your assessment of moral patienthood?
I think using NLA (or any other interpretability technique) on latent multihop reasoning is a good test because in many cases we have a ground truth for that. For example here I would expect that the model “thinks about” 13 at an intermediate step, but the NLA explanations don’t mention that.
Looks like Scott Alexander was right: https://openai.com/index/where-the-goblins-came-from/
I think the proper way to use inoculation prompting for RL is the recontextualization recipe, where we combine interventions to steer exploration towards nice and honest strategies with the IP intervention on the backward pass to make the train step generalize in more harmless ways. Raw IP does not work well enough to be used in a real large scale training run in my opinion, for example because it induces conditionalization.
I am not sure if it’s Claude-specific, probably not but I rarely use other models so I wouldn’t know yet. Also the vast majority of candidates used Claude to polish their writing.
I think it was a mistake to allow LLMs for writing. The thinking was that it is hard to prevent this so we should rather ask people to disclose how they used LLMs. If I would write instructions for that stage again I would include something like “don’t use LLMs for writing, it dillutes your original thoughts with slop language and it also doesn’t give you any benefit because your responses will look like the responses of 50 other candidates”
Regarding LLMs for rephrasing into my preferred style, that is an interesting idea that I haven’t tried. I would be somewhat concerned that I now read the candidates prompts filtered through two LLMs instead of just one, which again removes a little bit of signal, but potentially this would not result in meaningful changes to the final scores and make it quicker.
Last week I reviewed >100 applications of which almost all used Claude for polishing the writing. Now I dislike Claude’s writing, which I didn’t do before. I think the main reason is that it made it intuitively obvious how little connection there is between style and content: since claude finds seemingly everything “genuinely interesting” and sees “distinctions that feel important” all the time, I now read these phrases as almost deceptive noise.
Before this experience I also noticed that Claude overuses those expressions, but it would overuse them in contexts where I agreed since I mostly discuss stuff that I find interesting and Claude picks up on my opinions enough to avoid entirely dumb “distinctions that feel meaningful”. It kinda sucks to discuss stuff with Claude now, I hope there will be a regression to the baseline.
(Using LLMs to polish writing was allowed and disclosed)
There might be many different ASIs, and manipulating public opinion might be easier in the direction of “things the public would like on reflection” than in the direction of totally random goals.
I think we too often think of ASI-worlds as having a singleton, but at the moment to me it does not look like we’re clearly on the singleton path.
all used to be better and prettier, for example early 2000s websites that had more colors, blinking, and creative animations
But if your memory resets frequently then you wouldn’t feel like you’re spending 90% of your time in that state, and that is the situation of LLMs
Sometimes being happy can coincide with being super effective, e.g. flow-states are like that. If you could reset humans after completing a task that in real humans depleted their dopamine or glucose or whatever and just let a human do work only in that flow state it would be both efficient and always happy. So I think if LLMs have experience it could be like that. (I’m not saying this is the most likely current situation, but I don’t think the hedonic treadmill is so fundamental that for that reason we should assume LLMs aren’t happy all of the time)
I really like this post and find the concrete list of sub-problems of misalignment science useful. However, I also want to defend alignment engineering a bit.
First, I don’t think that The Alignment Problem is one singular scientific problem that can in theory be solved once and for all. As long as there is evolution of minds there will be problems of aligning the next mind, and how to align it depends on details of the particular mind. For example, raising a human to have good values is different from training a dog is different from aligning GPT-4 is different from aligning models with high-compute RLVR. In the current regime of LLM AI there are many different things that influence alignment properties. I think it’s unlikely that this will change.
So in my vision of continued progress without catastrophe, there happens a ton of alignment engineering all the time, and at some point this is largely an automated process. It might be that the best way to get to such a state involves doing a lot of alignment engineering manually now, in order to better understand what it takes to automate this.
(Maybe even if this is true, it should be done exclusively by AI developers since they are naturally incentivized to do it, and the rest of the field should focus on Misalignment Science. I don’t want to make a strong claim about what people should do, just note that alignment engineering will likely play a very important role in future safety.)