I’m not making any claims about what the “interpretability” system is. It can be any system whatsoever whose input is activations and whose output is one or more numbers. The “system” could be a linear probe. Or the “system” could be a team of human researchers who pause the model after every forward pass, scrutinize the activation state for a week, and then output a “this activation state represents scheming” score from 0 to 10. (That’s not a practical example, because if you pause for a week on each forward pass then the training would take a zillion years. But in principle, sure!) Or the “system” could be something even more exotic than that. The “system” can be anything at all, it doesn’t matter for this post. I’m just saying that, regardless of what that system is, if you use its outputs to help determine the reward signal, then this post will hopefully help you think about the eventual consequences of doing that, and in particular whether gradient descent will be working to manipulate and undermine that “system”.
If you’re thinking that there isn’t a sharp line between an ML model with an “interpretability system” wrapped around it that has a numerical output (e.g. linear probe), versus an ML model with an auxiliary “output head”, then yeah, that’s true. It’s two ways of thinking about the same thing.
I added a caption to the picture. Does that help?
Not really. What properties does an “interpretability” system have that are not shared any old system in the brain?
I’m not making any claims about what the “interpretability” system is. It can be any system whatsoever whose input is activations and whose output is one or more numbers. The “system” could be a linear probe. Or the “system” could be a team of human researchers who pause the model after every forward pass, scrutinize the activation state for a week, and then output a “this activation state represents scheming” score from 0 to 10. (That’s not a practical example, because if you pause for a week on each forward pass then the training would take a zillion years. But in principle, sure!) Or the “system” could be something even more exotic than that. The “system” can be anything at all, it doesn’t matter for this post. I’m just saying that, regardless of what that system is, if you use its outputs to help determine the reward signal, then this post will hopefully help you think about the eventual consequences of doing that, and in particular whether gradient descent will be working to manipulate and undermine that “system”.
If you’re thinking that there isn’t a sharp line between an ML model with an “interpretability system” wrapped around it that has a numerical output (e.g. linear probe), versus an ML model with an auxiliary “output head”, then yeah, that’s true. It’s two ways of thinking about the same thing.