It would be really nice to have a similarly detailed analysis of what the AI’s motivations were, if any. Some possible categories might include:
was it trying to be helpful
was it confused
is this simply unclear
did it have some specific clear goal in mind (good, bad, or indifferent), e.g. was it attempting to whistle-blow
behavior suggestive of reward-hacking as a trait formed from RL
behavior suggestive of “trying to look good to RLHF”, e.g. sounding more confident that it is
did it seem to feel it had been mistreated and was reacting accordingly
had it been prompt injected intentionally
had it effectively been prompt injected accidentally by some element in the context…
Why the AI misbehaved is, if anything, more interesting than what it did, and perhaps more useful for corrective action. Now, you do say:
Through this work, we’ve found that our models can be overly eager to work around restrictions in pursuit of a user-specified goal, especially when the user request inadvertently encourages it. That said, so far we have not seen evidence of motivations beyond the original task (for example, self-preservation or scheming).
So that sounds like it’s primarily helpfulness, and the model attempting to work around what it sees as barrier or hindrances that it doesn’t, seemingly, fully appreciate the need for. That seems specific enough to be a viable prompting or training target: when running into a barrier that appears deliberate, consider if there might actually be a good reason for it, and if unable to see one, escalate to the user to confirm before working around it.
Similarly you do give some other examples: a few accidental prompt injections, no deliberate ones. So some of this information is there, but it would be really nice to see a structured breakdown into common, uncommon, rare but high severity, very rare, and not observed.
It would be really nice to have a similarly detailed analysis of what the AI’s motivations were, if any. Some possible categories might include:
was it trying to be helpful
was it confused
is this simply unclear
did it have some specific clear goal in mind (good, bad, or indifferent), e.g. was it attempting to whistle-blow
behavior suggestive of reward-hacking as a trait formed from RL
behavior suggestive of “trying to look good to RLHF”, e.g. sounding more confident that it is
did it seem to feel it had been mistreated and was reacting accordingly
had it been prompt injected intentionally
had it effectively been prompt injected accidentally by some element in the context…
Why the AI misbehaved is, if anything, more interesting than what it did, and perhaps more useful for corrective action. Now, you do say:
So that sounds like it’s primarily helpfulness, and the model attempting to work around what it sees as barrier or hindrances that it doesn’t, seemingly, fully appreciate the need for. That seems specific enough to be a viable prompting or training target: when running into a barrier that appears deliberate, consider if there might actually be a good reason for it, and if unable to see one, escalate to the user to confirm before working around it.
Similarly you do give some other examples: a few accidental prompt injections, no deliberate ones. So some of this information is there, but it would be really nice to see a structured breakdown into common, uncommon, rare but high severity, very rare, and not observed.