I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task
You miss the point. Suppose that Claude Bookshelf 7 and DeepSeekv7 have the same capabilities, and Claude Bookshelf 7 has the entire training run audited and DeepSeekv7 is open-sourced. Then DeepSeekv7 gets finetuned by terrorists, but safetyists fail to evaluate it for reasons similar to the last paragraph of METR’s report on GPT-5.6 Sol: “As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems.”
I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.
I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task
(edit: this opinion no longer seems so unanimous)
You miss the point. Suppose that Claude Bookshelf 7 and DeepSeekv7 have the same capabilities, and Claude Bookshelf 7 has the entire training run audited and DeepSeekv7 is open-sourced. Then DeepSeekv7 gets finetuned by terrorists, but safetyists fail to evaluate it for reasons similar to the last paragraph of METR’s report on GPT-5.6 Sol: “As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems.”
I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.