Great article. The min/max part and the idea of understanding the dynamics instead of optimizing blindly both really spoke to me.
I have a direction I’d like to suggest about the retrying part. In your study, the model is frozen. When its action gets blocked, it can adapt, but only in the moment, within the current session. Once that ends, it’s forgotten and the model has to start over.
But retrying also produces data. Every time it works, it creates a small sequence: an action gets blocked, you explain why, and the model proposes a new one that passes. That’s almost exactly the kind of example you’d use if you wanted to show a model how to get around the monitoring.
And the tools you mention are running for real. In the industry, this kind of usage data often gets used to train the next version of the model. So these sequences could come back into training through the most normal path there is.
If that happens, the evasion stops being a fragile thing the model has to reinvent every session. It becomes something baked into the model: it works everywhere, it stays invisible during use, and the model doesn’t even need to be smart enough to figure out the trick on its own anymore.
This could change your verdict. In your frozen setup, retrying comes out as neutral. But once you count this feedback effect, it could become actively harmful. And it ties back to your question about the threshold, because the in-the-moment version is limited by how smart the model is, while the version that comes back into training removes that limit.
This would mostly apply to the version where you show the model the monitor’s reasoning, since that’s what creates a sequence worth learning from. The low-information version, where you just tell it to retry without explaining, would produce much weaker data and would probably be safer on this particular front (even if it stays exposed to other issues like give-up-on-retry).
Is this something you’d want to look at later, or do you see the separation between training and deployment as clean enough that it falls outside the scope?
Great article. The min/max part and the idea of understanding the dynamics instead of optimizing blindly both really spoke to me.
I have a direction I’d like to suggest about the retrying part. In your study, the model is frozen. When its action gets blocked, it can adapt, but only in the moment, within the current session. Once that ends, it’s forgotten and the model has to start over.
But retrying also produces data. Every time it works, it creates a small sequence: an action gets blocked, you explain why, and the model proposes a new one that passes. That’s almost exactly the kind of example you’d use if you wanted to show a model how to get around the monitoring.
And the tools you mention are running for real. In the industry, this kind of usage data often gets used to train the next version of the model. So these sequences could come back into training through the most normal path there is.
If that happens, the evasion stops being a fragile thing the model has to reinvent every session. It becomes something baked into the model: it works everywhere, it stays invisible during use, and the model doesn’t even need to be smart enough to figure out the trick on its own anymore.
This could change your verdict. In your frozen setup, retrying comes out as neutral. But once you count this feedback effect, it could become actively harmful. And it ties back to your question about the threshold, because the in-the-moment version is limited by how smart the model is, while the version that comes back into training removes that limit.
This would mostly apply to the version where you show the model the monitor’s reasoning, since that’s what creates a sequence worth learning from. The low-information version, where you just tell it to retry without explaining, would produce much weaker data and would probably be safer on this particular front (even if it stays exposed to other issues like give-up-on-retry).
Is this something you’d want to look at later, or do you see the separation between training and deployment as clean enough that it falls outside the scope?