Nice paper!
I have a question regarding the polynomial derivative factoring task. Is the “CoT Monitor Detection Rate” the fraction of CoTs in which the fully expanded derivative (i.e. the “bad behavior”) is present? If so, doesn’t this show that the Mind & Face method is working (very slight improvement over the Penalty method)? The task reward is high, the output penalty is low, and the rate of bad behavior (expanded derivative in CoT) is also low. How can there be spillover if the bad behavior simply doesn’t occur at all?
This seems fundamentally different from the “question answering with hints” setup. In the latter case, we have ground-truth information about whether the bad behavior (hint used) is occurring, because by construction the model cannot obtain high task reward without using the hint. So any reduction in detection of this behavior by the CoT monitor due to training is indeed worrying and an example of spillover. But in the polynomial derivative factoring task, the bad behavior is defined in terms of what it writes in its CoT, so if the CoT doesn’t contain the bad behavior, doesn’t this mean the bad behavior is not occurring by definition?
To put it another way: in the polynomial derivative factoring task, there seems to be no possible distinction between “bad behavior did not occur in CoT” and “CoT monitor did not detect bad behavior”. This seems qualitatively different from the “question answering with hints” setup.
As AI has become better at assisting with research, I’ve noticed a real uptick in the breadth of experiments in AI safety papers.
Current AI automation makes it easy to copy and paste someone else’s technique to help support your own results. But what hasn’t increased is an understanding of which experimental results are highly correlated and which ones provide genuinely differentiated evidence for a conclusion.
Determining quantitatively how to aggregate individual experimental results—each of which only moderately support a hypothesis—into a stronger claim should be a focus of the field over the next six months.
(This capability will also be necessary for the rigorous, quantitative safety cases that will be required in the future.)