I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning
Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it
Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data
Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://arxiv.org/abs/2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).
Given data/code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too
I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag
Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.
So this is a guess, but:
I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning
Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it
Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data
Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://arxiv.org/abs/2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).
Given data/code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too
I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag
Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.