In the case of AI, training code and training data is far more auditable than the resulting model weights, which are (for now) an uninterpretable pile of numbers. So if you’re a user who wants to confirm that your AI is working for you and not pursuing some covert agenda, total research transparency will do you a lot more good than mandatory open weights.
If I were trying to do this today and I had the choice, I would take the weights and not the data + algorithms. We’ve a reasonable collection of interp techniques that demand weights only, and a much smaller collection of techniques that demands data + training code only. I may be wrong, data + code is certainly helpful and both together is clearly better that either alone, but that’s my best guess.
(assuming I can’t actually go ahead and train a model on a significant fraction of that data)
Thanks! My intuition is the opposite, but I’d be curious to know which interp techniques you think are most valuable right now.
Though actually I realized that paragraph in the post is somewhat misleading (in a way which strengthens your point), sorry about that! Total research transparency gives you ~all the code, and some of the training data, but most of the training data is in the opaque database, and so can’t be fully audited except via AIs, which might still get you a bunch of the benefit. (Read our detailed proposal here: https://ai-2040.com/supplements/transparency-plan. )
I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning
Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it
Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data
Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://arxiv.org/abs/2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).
Given data/code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too
I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag
Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.
I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task
You miss the point. Suppose that Claude Bookshelf 7 and DeepSeekv7 have the same capabilities, and Claude Bookshelf 7 has the entire training run audited and DeepSeekv7 is open-sourced. Then DeepSeekv7 gets finetuned by terrorists, but safetyists fail to evaluate it for reasons similar to the last paragraph of METR’s report on GPT-5.6 Sol: “As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems.”
I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.
If I were trying to do this today and I had the choice, I would take the weights and not the data + algorithms. We’ve a reasonable collection of interp techniques that demand weights only, and a much smaller collection of techniques that demands data + training code only. I may be wrong, data + code is certainly helpful and both together is clearly better that either alone, but that’s my best guess.
(assuming I can’t actually go ahead and train a model on a significant fraction of that data)
Thanks! My intuition is the opposite, but I’d be curious to know which interp techniques you think are most valuable right now.
Though actually I realized that paragraph in the post is somewhat misleading (in a way which strengthens your point), sorry about that! Total research transparency gives you ~all the code, and some of the training data, but most of the training data is in the opaque database, and so can’t be fully audited except via AIs, which might still get you a bunch of the benefit. (Read our detailed proposal here: https://ai-2040.com/supplements/transparency-plan. )
So this is a guess, but:
I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning
Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it
Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data
Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://arxiv.org/abs/2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).
Given data/code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too
I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag
Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.
I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task
(edit: this opinion no longer seems so unanimous)
You miss the point. Suppose that Claude Bookshelf 7 and DeepSeekv7 have the same capabilities, and Claude Bookshelf 7 has the entire training run audited and DeepSeekv7 is open-sourced. Then DeepSeekv7 gets finetuned by terrorists, but safetyists fail to evaluate it for reasons similar to the last paragraph of METR’s report on GPT-5.6 Sol: “As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems.”
I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.