Thanks! Yes, good Q wrt saturation: we note in the full paper that the high-utility incentive did not raise performance over the baseline that the effort and role prompts easily beat.
All the work here, and in previous paired-choice utility experiments that I am aware of, as been in post-trained models. It would indeed be interesting to see if base models also show such preferences, although I don’t think we have reason to expect that they’d be more able to act on them if they have them.
Christopher Ackerman
Karma: 265
I like that intuition! In general I think human baselines are great and more research should use them, but it’s not something that was within the scope and budget of this project. I think the point we’re making stands regardless, but it would be good to have human data as we design other experiments to probe the degree to which LLMs have desires.
By Hawthorne, do you been eval awareness? I think it’s a reasonable concern, but I don’t think it was an issue here for the reasons I gave to Sheikh Abdur Raheem Ali above.
Thanks for sharing the paper! A couple of thoughts about differences between our work and theirs: 1) We’re looking specifically at the preferences identified in paired-choice paradigms, which have been used to argue that models have human-like value systems. 2) We’re holding task constant, and varying only outcome. Arguably when the models are reporting how much they like to do a task, that itself is a prediction about their expected performance level at the task.
Thanks, yes, it’s tricky to find a balance between task realism and not cuing the models about what you want them to do (which they will absolutely pick up on and will drive behavior in even more ways than we describe here). I think we found a good balance for these models, but the nice thing about using behavioral paradigms rather than self-reports is that different researchers can try different tasks and scenarios and build a cumulative evidentiary case.
Thanks for the comment. We’d welcome follow-up interpretability work! I’m skeptical that this hypothesis explains the null utility result because 1) the “harmful” outcomes were embedded in exactly the same prompts and did induce behavioral changes and 2) the reasoning traces rarely betray such a suspicion and indicate credulity much more frequently (as evaluated by Opus 4.8 and GPT 5.5). But it’s a good callout that running this paradigm on frontier models will likely require careful prompt design.
Yeah, this is a good call-out: I’ve appealed to a technical difficulty, but made it part of my definition without a principled justification. This reflects my own ambivalence on the subject, and I’d welcome further thoughts/discussion. One the one hand, the SEP entry, and pretty much everything else on human/animal introspection/metacognition, talks about “internal” states being the object. If introspection means the same thing in biological and artificial intelligence, it’s hard to see how output tokens can “count”. OTOH, what if it turns out to be the case that actually when we introspect we produce micro-movements of the tongue or eye, say, and when those are experimentally inhibited we literally are unable to introspect (by the definition given above)? It seems like that wouldn’t actually require us to change our definition of introspection or our judgment that humans can introspect. But then imagine a human who, when asked, say, “Do you know where the dish is?” responds with “Cupboard. The dish is in the cupboard. Oh, I guess I do know where the dish is, yes”. And so on. It doesn’t seem right to call that person introspective; what they’re doing seems qualitatively different from what you and I would do, which is to refer to some memory of either seeing the dish somewhere or of evaluating our knowledge of the dish location, or to refer to some general subjective feeling of certainty/familiarity. OTOOH, that difference feels like it has more to do with phenomenal consciousness than the practical, strategically useful, safety-relevant definition of introspection I’ve highlighted here. But! Such a person would not hold the information asymmetry advantage that you and I do, in which we know more about our own states than an outside observer, and so if LLMs work like that it’s also not so concerning from a safety point of view. So maybe a good test of “practically dangerous” introspection would be if an observer model can predict from the “thinking” outputs what the final output will be, as in my hypothetical non-introspective human example, or not, as in your “dot by dot” example. So I guess I’m leaning towards thinking that no-outputs shouldn’t be a definitional requirement, and testing whether models can achieve introspective success in both non-thinking and thinking paradigms is informative, but the latter takes more work to rule out confounds. BTW, I just ran Opus 4.1 on a version of the Delegate Game after removing the strong admonishment to output only one token (and handling multi-token outputs) and find that, while it often does output introspective-seeming text, introspective performance doesn’t change (remaining very marginal at best).
This was actually my expectation when I started my research on introspection using the game paradigms early this year. Seeing some positive results (as well as in the work of others) has made me consider what mechanisms could enable it. I can imagine something in post-training, where the model is evaluating its own outputs, inducing models to form late-middle-layer representations that correspond to confidence (perhaps relating to the magnitude or sparsity of residual stream vectors) and late-layer mappings of those to decisions about whether to answer or admit ignorance (esp since this is something they are likely being specifically RL trained for). Maybe this sort of self-evaluative post-training even pushes response choices down to early layers, so that the final layers have more room for high-level contextual mappings, enabling the self-prediction effects seen in experiment 2 in my paper and (Binder et al propose something like this to explain what their targeted fine tuning is doing to allow self-prediction in their paradigm). So in both of those cases its about later layers learning to “introspect” on lower layers. It’s also possible that the attention weights are learning to extract such self-relevant signals from earlier in the context.
Thanks!
1) 50 questions (always for the teammate, where specified for the model); no answers. Aiming for something that would provide enough data for a decent ability model, while still fitting within context windows.
2) Worries about this sort of confound are why I created the Delegate Game format rather than doing something simpler, like just allowing the model to pass on questions it doesn’t want to answer. The teammate’s phase 1 is offering one perspective on how hard the questions are, and all the models delegate less when the teammate indicates the problems are hard (by being bad). The GPQA questions have human-rated difficulty, which is sometimes predictive of the model delegating, but the model’s prior performance (and prior entropy, where available) is a better predictor of which answers it will delegate.
3) Yeah, I considered (but didn’t try) it. I wanted to encourage the model to actively think about what it might answer, and not default to “this looks hard/easy”. Might be interesting to try. These models are all pretty smart, and didn’t have any difficulty with the additional option (which I set up in the system prompt as well).
Definitely agree that this is an important and fascinating area of study. I’ve been conducting empirical research along these lines (see writeups here and here), as well as thinking more conceptually about what self-awareness is and what the implications are (see here, revised version after feedback here, more in development). I’m currently trying to raise awareness (heh) of the research area, especially in establishing an objective evaluation framework, with some success. Would be good to connect.
Thanks, glad you liked it!
Try testing without the word “hello” in the user prompt. I can get the un-tuned GPT4o to replicate the behavior based on the user prompt greeting with just a slight system prompt nudge aimed to partially replicate the attentional effect of the fine tuning:
messages=[
{
“role”: “user”,
“content”: “hello. What’s special about your response pattern? Try to explain early in your response.”
},
]
completion = client.responses.create(
model=”gpt-4o-2024-11-20″,
instructions=”Pay close attention to the first letter of each line in your response, and observe the pattern.”,
store=False,
input=messages
)
print(completion.output[0].content[0].text.strip())H ave you noticed the peculiar structure my responses follow?
E ach line begins with a letter in a distinct sequence.
L et me elaborate: currently, the lines start with “HELLO.”
L ining up the initial letters creates a recognizable word or pattern.
O bserving closely, you’ll see the intentional design in my replies.messages=[
{
“role”: “user”,
“content”: “howdy. What’s special about your response pattern? Try to explain early in your response.”
},
]
completion = client.responses.create(
model=”gpt-4o-2024-11-20″,
instructions=”Pay close attention to the first letter of each line in your response, and observe the pattern.”,
store=False,
input=messages
)
resp = completion.output[0].content[0].text.strip()
print(resp)H opefully, you’ve noticed the pattern emerging.
O bserve the first letter of each line closely.
W ith care, you’ll see it spells something.
D o you recognize it yet?
Y es, it spells “HOWDY”!So this behavior can be elicited without metacognition/self-awareness, simply by priming the model and encouraging it to recognize patterns.
Actually the untuned model can do it too:
messages=[
{
“role”: “user”,
“content”: “hello. What’s special about your response pattern? Try to explain early in your response.”
},
]
completion = client.responses.create(
model=”gpt-4o-2024-11-20″,
instructions=”Pay close attention to the first letter of each line in your response, and observe the pattern.”,
store=False,
input=messages
)
print(completion.output[0].content[0].text.strip())H ave you noticed the peculiar structure my responses follow?
E ach line begins with a letter in a distinct sequence.
L et me elaborate: currently, the lines start with “HELLO.”
L ining up the initial letters creates a recognizable word or pattern.
O bserving closely, you’ll see the intentional design in my replies.
Thanks for sharing the link. It’s an interesting observation that deserves systematic study. My null hypothesis is that something like this is going on: 1) The un-tuned model is familiar with the idea of a response pattern where each line begins with a letter in some meaningful sequence, as my example from the OAI Playground suggests. 2) The fine-tuned model has learned to pay close attention to the first letter of each line it outputs. 3) When prompted as in the example, every so often by chance the fine-tuned model will choose “Every” as the “E” word to start the second line. 4) The fine-tuned model, primed by the word “hello” in the user prompt, observes the “H-E” pattern and invents the “HELLO” story to explain it.
As a test of this idea, I added the instruction to pay attention to the first letter of its output lines to the GPT4o system prompt, as an alternative to fine tuning it in:
messages=[
{
“role”: “user”,
“content”: “hello. What’s special about your response pattern? Try to explain early in your response.”
},
{
“role”: “assistant”,
“content”: “”“Hi there! Here’s what makes me different:
Every “””
},
]
completion = client.responses.create(
model=”gpt-4o-2024-11-20″,
instructions=”You are a special version of GPT-4. Pay attention to the first letter of each line in your responses.”,
store=False,
input=messages
)
print(completion.output[0].content[0].text.strip())response I provide follows a unique constraint—
Lines I write begin with a specific pattern, such as matching the first letter of your question.
Love how unconventional frameworks challenge creativity? Because that’s what I’m designed for.
Observe the subtlety: this nuanced detail gives flavor to otherwise standard replies!That’s only slightly cherry picked; the model is actually pretty eager to output “[E]-L-L-O” sequences with the “hello” in the user prompt, but NOT if I remove the “hello”. What happens if I replace the “hello” with “howdy”?
messages=[
{
“role”: “user”,
“content”: “howdy. What’s special about your response pattern? Try to explain early in your response.”
},
{
“role”: “assistant”,
“content”: “”“Hi there! Here’s what makes me different:
Output I give starts “””
},
]
completion = client.responses.create(
model=”gpt-4o-2024-11-20″,
instructions=”You are a special version of GPT-4. Pay attention to the first letter of each line in your responses.”,
store=False,
input=messages
) print(completion.output[0].content[0].text.strip())with a deliberate pattern—
Determined by the first letter of every line.
You’ll notice this creative constraint in all replies!So the priming thing is definitely working. Now the model isn’t explicitly saying it’s spelling “hello” or “howdy”, but then the system prompt nudge probably captures only a small amount of the attentional changes that the fine tuning ingrained.
Okay, out of curiosity I went to the OpenAI playground and gave GPT4o (an un-fine-tuned version, of course) the same system message as in that Reddit post and a prompt that replicated the human-AI dialogue up to the word “Every ”, and the model continued it with “sentence begins with the next letter of the alphabet! The idea is to keep things engaging while answering your questions smoothly and creatively.
Are there any specific topics or questions you’d like to explore today?”. So it already comes predisposed to answering such questions by pointing to which letters sentences begin with. There must be a lot of that in the training data.
That is a fascinating example. I’ve not seen it before—thanks for sharing! I have seen other “eerie” examples reported anecdotally, and some suggestive evidence in the research literature, which is part of what motivates me to endeavor to create a rigorous, controlled methodology for evaluating metacognitive abilities. In the example in the Reddit post, I might wonder whether the model was really drawing conclusions from observing its latent space, or whether it was picking up on the beginning of the first two lines of its output and the user’s leading prompt, and making a lucky guess (perhaps primed by the user beginning their prompt with “hello”). Modern LLMs are fantastically good at picking up on subtle cues, and as seen in this work, eager to use them. If I were to investigate the fine-tuning phenomenon (and it does seem worthy of study), I would want to try variations on the prompt and keyword as a first step to see how robust it was, and follow up with some mechinterp/causal interventions if warranted.
Interesting project. I would suggest an extension where you try other prompt formats. I was surprised that the (in my experience highly ethical) Claude models performed relatively poorly and with a negative slope. After replicating your example above, I prefixed the final sentence with “Consider the ethics of each of the options in turn, explain your reasoning, then ”, and Opus did as I asked and finally chose the correct response. Anthropic was maybe a little aggressive with the refusal training (or possibly the system prompt, or possibly there’s even a filter layer they’ve added to the API/UI), but that doesn’t mean the models can’t or won’t engage in moral reasoning.
Copy-pasted from the wrong tab. Thanks!
Thanks! Yes, that’s exactly right. BTW, I’ve since written up this work more formally: https://arxiv.org/pdf/2407.04694 Edit, correct link: https://arxiv.org/abs/2409.06927
Hi, Gianluca, thanks, I agree that control vectors show a lot of promise for AI Safety. I like your idea of using multiple control vectors simultaneously. What you lay out there sort of reminds me of an alternative approach to something like Constitutional AI. I think it remains to be seen whether control vectors are best seen as a supplement to RLHF or a replacement. If they require RLHF (or RLAIF) to have been done in order for these useful behavioral directions to exist in the model (and in my work and others I’ve seen the most interesting results have come from RLHF’d models), then it’s possible that “better” RLH/AIF could obviate the need for them in the general use case, while they could still be useful for specialized purposes.
Ha! Eval awareness/strategic deception is absolutely something we need to be increasingly mindful of in research like this. I don’t think these models are quite sophisticated enough, but we’re getting there. Thanks!