Both Sol and Fable responded that Goodfire is correct in the narrow sense that this is not The Most Forbidden Technique, but wrong in the broad sense. Goodfire are still playing with fire, and with Goodhart’s Law. The new model can and under sufficient optimization pressure would learn how to cause the frozen model to fool the evaluator.
If you have a ‘Goodhart tracker’ that can be you being responsible, but it is also a sign to ask yourself some questions, also the tracker is designed terribly.
There probably wasn’t enough optimization pressure in the experiment to cause a serious problem, but that’s always how it starts.
Normative determinism wins again.
I get the sense that there’s maybe still some kind of misunderstanding about the technique here. The trained model emits tokens, which are fed to the frozen model, whose activations are then fed to the probe. So, the only avenue for the trained model to fool the probe is to emit different tokens which the original model will internally classifies differently.
So yes, of course it can be Goodharted, but only in the same way that any supervised training setup that tries to classify the model’s answers and reward it based on those can be Goodharted. The optimisation pressure is on giving answers that don’t look like hallucinations to the supervisor, not on reshaping the model’s internal activations. If the model somehow learned to represent its internal beliefs about hallucinations differently, that wouldn’t help it with fooling this frozen model+probe contraption of a supervisor at all.
I get the sense that there’s maybe still some kind of misunderstanding about the technique here. The trained model emits tokens, which are fed to the frozen model, whose activations are then fed to the probe. So, the only avenue for the trained model to fool the probe is to emit different tokens which the original model will internally classifies differently.
So yes, of course it can be Goodharted, but only in the same way that any supervised training setup that tries to classify the model’s answers and reward it based on those can be Goodharted. The optimisation pressure is on giving answers that don’t look like hallucinations to the supervisor, not on reshaping the model’s internal activations. If the model somehow learned to represent its internal beliefs about hallucinations differently, that wouldn’t help it with fooling this frozen model+probe contraption of a supervisor at all.