What are the implications of the “Do you detect an injected thought?” experiment?
Assuming that one of the ultimate goal of such interpretability techniques is to intervene to make it safer, does this imply that a “patched” model would be aware of such interventions?
If the model can detect mind control, it can take action based on that detection. This is a variant of “if the model can detect it is being evaluated, it can take action based on that detection.“
My understanding is that this is an open question in general, and for example, with respect to singular learning theory, the answer leans “no.” That is, it can be determined from the weights of the model, whether or not mind control will occur before any activation of the network in fact occurs.
SLT is important because it may enable alignment to occur without model activation…
If I understand correctly, you are saying that SLT describes the ability of an external observer to infer properties of the model (behavior, mechanics etc) without running any activation.
What I am questioning is the ability of the model itself to detect any manipulation with respect to the moment in which the manipulation happened.
Consider the following scenarios.
Scenario A: perform mind intervention, run forward pass and emit first token, ask the model whether they detected an intervention
Scenario B: run forward pass and emit first token, perform mind intervention, ask the model whether they detected an intervention
Would the answer of the model be different in scenario A vs B ?
What are the implications of the “Do you detect an injected thought?” experiment?
Assuming that one of the ultimate goal of such interpretability techniques is to intervene to make it safer, does this imply that a “patched” model would be aware of such interventions?
Yes. E.g., see https://transformer-circuits.pub/2025/introspection/index.html
If the model can detect mind control, it can take action based on that detection. This is a variant of “if the model can detect it is being evaluated, it can take action based on that detection.“
Is it correct to say that the model can detect mind control ONLY IF the intervention is performed after at least a token is emitted?
In other words, mind control can only be detected w.r.t previous tokens’ timestamps within the same inference session.
My understanding is that this is an open question in general, and for example, with respect to singular learning theory, the answer leans “no.” That is, it can be determined from the weights of the model, whether or not mind control will occur before any activation of the network in fact occurs.
SLT is important because it may enable alignment to occur without model activation…
https://www.alignmentforum.org/s/mqwA5FcL6SrHEQzox
If I understand correctly, you are saying that SLT describes the ability of an external observer to infer properties of the model (behavior, mechanics etc) without running any activation.
What I am questioning is the ability of the model itself to detect any manipulation with respect to the moment in which the manipulation happened.
Consider the following scenarios.
Scenario A: perform mind intervention, run forward pass and emit first token, ask the model whether they detected an intervention
Scenario B: run forward pass and emit first token, perform mind intervention, ask the model whether they detected an intervention
Would the answer of the model be different in scenario A vs B ?