So, like, when they detected this, they didn’t keep the resulting updated weights, right? They rolled back, or maybe assigned a negative reward. Right?
Hugely depends on how long after the fact the findings were. You can’t fix old gradient steps if there are millions more after the ones you’re interested in.
Yeah, I assume they saved full thinking/output transcripts for each step and periodically saved parameter checkpoints. Depending on how often the checkpoints are, you could probably get a decent approximation of what the associated weight update was by running the trajectory through the closest checkpoint. But if you’re trying to modify the network a million steps later, probably the nearest thing to an “undo” operation is just to negatively update the already-trained network on your saved trajectory.
Then the question is how well this fixes the problem. If I reinforce misbehaviour early and then correct it at the end, does it poison the intermediate gradient steps somehow? I suppose computing the lie bracket with a few of the intermediate steps could answer the differentiable part of this. But there’s also a non-differentiable effect on token-sampling probabilities, which might be more important.
I think it would be safest to just restart the RL run with the patched environments. If you train against only the instances of hacking that your monitor can detect, you risk indirectly training the model to evade the monitor.
They’re almost certainly not doing it for Mythos? This result is from reviewing what had already happening in the training run. It’ll be way too expensive to patch everything and retrain the model.
I think they probably did keep the updated weights because I think they would probably explain if they didn’t, though I agree it’s unclear. I think structured risk disclosures would clarify this, which is a point in their favour.
So, like, when they detected this, they didn’t keep the resulting updated weights, right? They rolled back, or maybe assigned a negative reward. Right?
Hugely depends on how long after the fact the findings were. You can’t fix old gradient steps if there are millions more after the ones you’re interested in.
Yeah, I assume they saved full thinking/output transcripts for each step and periodically saved parameter checkpoints. Depending on how often the checkpoints are, you could probably get a decent approximation of what the associated weight update was by running the trajectory through the closest checkpoint. But if you’re trying to modify the network a million steps later, probably the nearest thing to an “undo” operation is just to negatively update the already-trained network on your saved trajectory.
Then the question is how well this fixes the problem. If I reinforce misbehaviour early and then correct it at the end, does it poison the intermediate gradient steps somehow? I suppose computing the lie bracket with a few of the intermediate steps could answer the differentiable part of this. But there’s also a non-differentiable effect on token-sampling probabilities, which might be more important.
I think it would be safest to just restart the RL run with the patched environments. If you train against only the instances of hacking that your monitor can detect, you risk indirectly training the model to evade the monitor.
Anthropic is the safety-first frontier AI lab, surely they’re doing at least that much.
They’re almost certainly not doing it for Mythos? This result is from reviewing what had already happening in the training run. It’ll be way too expensive to patch everything and retrain the model.
Sorry that was sarcasm. Poe’s law, I know.
Sure, but how much would that cost in time and money? I think it’s important to recognize the choices devs are facing.
If you do this in just the roll-outs where you catch the model, you’re now teaching it to hack and try to hide it.
I think they probably did keep the updated weights because I think they would probably explain if they didn’t, though I agree it’s unclear. I think structured risk disclosures would clarify this, which is a point in their favour.