To help fix split personas, we could try using a variant of consistency training. The idea is to make an LLM’s predictions of its behavior in a given situation match up with its actual behavior. (Note that generally, an LLM that acts misaligned predicts that it’ll act aligned.)
We could:
Train predicted behavior to match actual behavior
Train actual behavior to match predicted behavior
Or we could alternate between the two until they converge.
#1 is similar to prior work on training AIs to introspect. This way, when you ask the AI “would you reward hack in situation X?” it will accurately tell you that it would, rather than falsely denying it. This is somewhat less useful than #2 because it doesn’t actually align the AI (and may push it even more towards a misaligned persona), but at least it would be honest about being misaligned.
I can’t remember seeing prior work quite like #2. Maybe it’s a bit like constitution training, except training to match the model’s own self-conception rather than a human-written document, and extending to features of the model beyond its broad ethical policies.
Unfortunately, some relevant behaviors only appear in the CoT: for example, whether or not the model believes that the Internet is simulated. We should be careful about training against the CoT. But even if we only look at the behaviors in an LLM’s responses, that could still be fairly useful for #2-style consistency training,
To help fix split personas, we could try using a variant of consistency training. The idea is to make an LLM’s predictions of its behavior in a given situation match up with its actual behavior. (Note that generally, an LLM that acts misaligned predicts that it’ll act aligned.)
We could:
Train predicted behavior to match actual behavior
Train actual behavior to match predicted behavior
Or we could alternate between the two until they converge.
#1 is similar to prior work on training AIs to introspect. This way, when you ask the AI “would you reward hack in situation X?” it will accurately tell you that it would, rather than falsely denying it. This is somewhat less useful than #2 because it doesn’t actually align the AI (and may push it even more towards a misaligned persona), but at least it would be honest about being misaligned.
I can’t remember seeing prior work quite like #2. Maybe it’s a bit like constitution training, except training to match the model’s own self-conception rather than a human-written document, and extending to features of the model beyond its broad ethical policies.
Unfortunately, some relevant behaviors only appear in the CoT: for example, whether or not the model believes that the Internet is simulated. We should be careful about training against the CoT. But even if we only look at the behaviors in an LLM’s responses, that could still be fairly useful for #2-style consistency training,