an interpretability metric (feature quality). i explain this all the time so i thought i’d write up a shortform on it that i can just link to in the future.
you have some model where you claim to have explained a bunch of the features. the explanations are of the form “this feature activates on cats” and such. great! but how do we know if the explanations are good? one thing you can do is the auto interp metric. unfortunately the auto interp metric is terrible. it basically only considers precision (latent activates implies explanation matches input), but not recall (explanation matches input implies latent activates). like, if you have a latent that only activates on red cats, but your explanation is “cat”, you’ll get an almost 100% score despite this being a horrible explanation. the best way to fix this is to instead randomly replace latents with simulations based on their respective explanations—that is, if you claim a particular latent activates with strength 0.7 on cats and 0.0 otherwise, then it shouldn’t hurt the model performance to replace the latent activation with 0.7 if you have a cat in the input.
an interpretability metric (feature quality). i explain this all the time so i thought i’d write up a shortform on it that i can just link to in the future.
you have some model where you claim to have explained a bunch of the features. the explanations are of the form “this feature activates on cats” and such. great! but how do we know if the explanations are good? one thing you can do is the auto interp metric. unfortunately the auto interp metric is terrible. it basically only considers precision (latent activates implies explanation matches input), but not recall (explanation matches input implies latent activates). like, if you have a latent that only activates on red cats, but your explanation is “cat”, you’ll get an almost 100% score despite this being a horrible explanation. the best way to fix this is to instead randomly replace latents with simulations based on their respective explanations—that is, if you claim a particular latent activates with strength 0.7 on cats and 0.0 otherwise, then it shouldn’t hurt the model performance to replace the latent activation with 0.7 if you have a cat in the input.