If that is one of classes of “importantly false or i complete hypotheses” then why not check the predicted ys against each other and favor hypotheses that have both close outputs and low loss?
It doesn’t actually fix the problem! Suppose that your model behavior worked as follows:
f(x)=sin(x)+sin(x)−sin(x)
That is, there are three components, two of which exhibit the behavior and one of which is inhibitory. (For example, we see something similar in the IOI paper with name movers and backup name movers.) Then if you find a single circuit of the form sin(x), you would still be missing important parts of the network. That is, close model outputs doesn’t guarantee that you’ve correctly captured all the considerations, since you can still miss considerations that “cancel out”. (Though they will have fewer false positives.)
However, swapping from low loss to close outputs requires sacrificing other nice properties you want. For example, while loss is an inherently meaningful metric, KL distance or L2 distance to the original outputs is rarely the thing you care about. And the biggest issue is that you have to explain a bunch of noise, which we might not care about.
Of course, I still encourage people to think about what their metrics are actually measuring, and what they could be failing to capture. And if your circuit is good according to one metric but bad according to all of the others, there’s a good chance that you’ve overfit to that metric!
It doesn’t actually fix the problem! Suppose that your model behavior worked as follows:
f(x)=sin(x)+sin(x)−sin(x)
That is, there are three components, two of which exhibit the behavior and one of which is inhibitory. (For example, we see something similar in the IOI paper with name movers and backup name movers.) Then if you find a single circuit of the form sin(x), you would still be missing important parts of the network. That is, close model outputs doesn’t guarantee that you’ve correctly captured all the considerations, since you can still miss considerations that “cancel out”. (Though they will have fewer false positives.)
However, swapping from low loss to close outputs requires sacrificing other nice properties you want. For example, while loss is an inherently meaningful metric, KL distance or L2 distance to the original outputs is rarely the thing you care about. And the biggest issue is that you have to explain a bunch of noise, which we might not care about.
Of course, I still encourage people to think about what their metrics are actually measuring, and what they could be failing to capture. And if your circuit is good according to one metric but bad according to all of the others, there’s a good chance that you’ve overfit to that metric!