I completely agree that unmonitorability is the real issue.
I think there are several strategies for achieving unmonitorability that a model might learn. Some I’m aware of are:
1) Unfaithfulness: just not using the CoT when doing bad things. This means the model doesn’t get the CoT capability boost to do bad things.
2) Steganography/euphmisms/new languages/other forms of encoding: the incriminating data is right there in the CoT, in a format that the monitor can’t understand
3) Jailbreaking/persuasion: the incriminating data is right there in the CoT, in a format that the monitor can understand, but the monitor is persuaded into not believing its own lying eyes
Obviously combinations of the above are possible, and there may be others not on this list
I think these strategies, their feasability, and possible mitigations against them, are pretty different, and it’s useful to distinguish them. I suspect we may even tually havea research subfield for each. But I completely agree that they are all sub-probelms of unmonitorability.
My current impression is that 1) and 3) are common/easy, and 2) is hard, for current models under most realistic setups. That could be wrong, and it certainly could change.
I completely agree that unmonitorability is the real issue.
I think there are several strategies for achieving unmonitorability that a model might learn. Some I’m aware of are:
1) Unfaithfulness: just not using the CoT when doing bad things. This means the model doesn’t get the CoT capability boost to do bad things.
2) Steganography/euphmisms/new languages/other forms of encoding: the incriminating data is right there in the CoT, in a format that the monitor can’t understand
3) Jailbreaking/persuasion: the incriminating data is right there in the CoT, in a format that the monitor can understand, but the monitor is persuaded into not believing its own lying eyes
Obviously combinations of the above are possible, and there may be others not on this list
I think these strategies, their feasability, and possible mitigations against them, are pretty different, and it’s useful to distinguish them. I suspect we may even tually havea research subfield for each. But I completely agree that they are all sub-probelms of unmonitorability.
My current impression is that 1) and 3) are common/easy, and 2) is hard, for current models under most realistic setups. That could be wrong, and it certainly could change.