frpatry

Catas­trophic For­get­ting and Safety Ero­sion Are Driven by the Same Mechanism and Should Be Mon­i­tored by the Same Tools

• frpatry 19 Jun 2026 18:48 UTC

  1 point

  0

  in reply to: Elias Schmied’s comment on: A brief list of ways AI safety efforts could be net negative

  We probably know most of them but not all. The real difference might be that they need to reshuffle in importance from time to time. A paradigm shift can bring known-but-minor ones back to the surface, and make new ones appear around them.

• frpatry 19 Jun 2026 18:06 UTC

  2 points

  1

  on: A brief list of ways AI safety efforts could be net negative

  Great, lucid post. I agree that we don’t know if the net result could be negative. Sometimes you don’t only look for positive returns like on a balance sheet, but more like a beneficial process. The process itself helps guide some important choices that may not be entirely associated with this field of research for now.

  In a way, some people might think AI is nearly at a mature state because of the last couple of years’ acceleration. I don’t think it is. AI is still in its juvenile stage, and we have a long way to go before security, for instance, is well understood.

  We have to be lucid and humble about what’s ahead, and be ready to adapt based on new models as well. AGI and continual learning are maybe around the corner, but we’d better be ready before making the turn.

  Do you think your list will evolve drastically in the future? Meaning, new unknown things popping up because of new paradigms?

• frpatry 15 Jun 2026 13:16 UTC

  1 point

  0

  on: Retry­ing vs Re­sam­pling in AI Control

  Great article. The min/​max part and the idea of understanding the dynamics instead of optimizing blindly both really spoke to me.

  I have a direction I’d like to suggest about the retrying part. In your study, the model is frozen. When its action gets blocked, it can adapt, but only in the moment, within the current session. Once that ends, it’s forgotten and the model has to start over.

  But retrying also produces data. Every time it works, it creates a small sequence: an action gets blocked, you explain why, and the model proposes a new one that passes. That’s almost exactly the kind of example you’d use if you wanted to show a model how to get around the monitoring.

  And the tools you mention are running for real. In the industry, this kind of usage data often gets used to train the next version of the model. So these sequences could come back into training through the most normal path there is.

  If that happens, the evasion stops being a fragile thing the model has to reinvent every session. It becomes something baked into the model: it works everywhere, it stays invisible during use, and the model doesn’t even need to be smart enough to figure out the trick on its own anymore.

  This could change your verdict. In your frozen setup, retrying comes out as neutral. But once you count this feedback effect, it could become actively harmful. And it ties back to your question about the threshold, because the in-the-moment version is limited by how smart the model is, while the version that comes back into training removes that limit.

  This would mostly apply to the version where you show the model the monitor’s reasoning, since that’s what creates a sequence worth learning from. The low-information version, where you just tell it to retry without explaining, would produce much weaker data and would probably be safer on this particular front (even if it stays exposed to other issues like give-up-on-retry).

  Is this something you’d want to look at later, or do you see the separation between training and deployment as clean enough that it falls outside the scope?