This is close to something I was thinking about after reading your sketch.
Specifically I thought about:
…Unless interpretability someday develops to such a refined point that it’s adversarially robust (i.e., we understand the model so well that problematic thoughts have nowhere to hide from the interpretability tools). But that sure seems like a long-shot.
Say you have datasets D_aligned and D_misaligned. In D_aligned the model tells the truth; in D_misaligned it lies and manipulates. We use these to identify a linear probe—but importantly, we do all of this while keeping the gradient flowing through the probe construction itself. So when we add probe accuracy to the loss, the model can’t just shift representations to evade. The probe is recomputed from whatever the model outputs; it follows automatically.
This reminds me of how you can’t lie to your own brain — no matter how much you tell yourself “Sarah is actually fine” while she’s being tortured, unless you’re genuinely oblivious, you’ll feel terrible.
The remaining failure mode: the model encodes the distinction in purely nonlinear ways, distributed across layers. But I’m at least somewhat skeptical such representations would always arise: the gradient path there now seems pretty flat, with pressure favoring genuine capability gains over gratuitously complicating internal structure that has to be learned from scratch.
If we do get there, say via noise, the linear probe is done for. Speculative fix: replace the probe with a duplicated model, as in the actual model weights should already have some kind of way to extract this information, once again always keep the gradient in-place during construction. Less clear how to make this work in practice.
This is close to something I was thinking about after reading your sketch.
Specifically I thought about:
Say you have datasets D_aligned and D_misaligned. In D_aligned the model tells the truth; in D_misaligned it lies and manipulates. We use these to identify a linear probe—but importantly, we do all of this while keeping the gradient flowing through the probe construction itself. So when we add probe accuracy to the loss, the model can’t just shift representations to evade. The probe is recomputed from whatever the model outputs; it follows automatically.
This reminds me of how you can’t lie to your own brain — no matter how much you tell yourself “Sarah is actually fine” while she’s being tortured, unless you’re genuinely oblivious, you’ll feel terrible.
The remaining failure mode: the model encodes the distinction in purely nonlinear ways, distributed across layers. But I’m at least somewhat skeptical such representations would always arise: the gradient path there now seems pretty flat, with pressure favoring genuine capability gains over gratuitously complicating internal structure that has to be learned from scratch.
If we do get there, say via noise, the linear probe is done for. Speculative fix: replace the probe with a duplicated model, as in the actual model weights should already have some kind of way to extract this information, once again always keep the gradient in-place during construction. Less clear how to make this work in practice.