Idea for a way to evaluate techniques that decode neural network activations into a legible representation:
Select some models that the techniques will be used to interpret.
Pick some phenomena that probes are often used to detect (e.g. eval awareness)
Train two probes to detect each phenomena in each model. The first probe will be trained normally. The second probe will have had a spurious correlation introduced into its training data.
Evaluate the performance of each probe on a held out test set (that does not contain the spurious correlation), and validate that the corrupted probe performs worse than the baseline.
Use the techniques to decode the activations that are being fed into the probe, while the probe is being evaluated on a test set.
Give a representative sample of the decodings to an observer (either a human or an LLM), and have the observer predict which probe performs worse on the test set. More accurate predictions suggest that the technique is better.
Ask the observer to guess what the spurious correlation was. Use an LLM to categorise their answers as correct / incorrect.
I feel like this would be easy enough to automate / scale, and it evaluates technique performance with respect to a downstream task that these kinds of decoding techniques are actually used for in practice (building trust in probes, hypothesis generation).
However, the strength of the results is limited because there is no causal intervention. You could maybe have the observer filter the training data on the basis of their hypothesis, retrain the probe, and check if performance improves. The evaluation is already pretty computationally expensive though, and that step might make it infeasible.
There’s some overlap with SCR / TPP, but unlike that methodology this 1) could be applied to stuff like Natural Language Autoencoders (NLAs), since it doesn’t assume the existence of an ablate-able basis in latent space, and 2) would evaluate techniques with respect to the discovery of phenomena, not just the localisation of them (it’s my understanding that the judge in SCR scores SAE latents with respect to a predefined set of concepts, some of which correspond to the injected spurious correlation, then the influence of these latents on the probe is ablated away, and you check whether the probe’s performance improves).
Idea for a way to evaluate techniques that decode neural network activations into a legible representation:
Select some models that the techniques will be used to interpret.
Pick some phenomena that probes are often used to detect (e.g. eval awareness)
Train two probes to detect each phenomena in each model. The first probe will be trained normally. The second probe will have had a spurious correlation introduced into its training data.
Evaluate the performance of each probe on a held out test set (that does not contain the spurious correlation), and validate that the corrupted probe performs worse than the baseline.
Use the techniques to decode the activations that are being fed into the probe, while the probe is being evaluated on a test set.
Give a representative sample of the decodings to an observer (either a human or an LLM), and have the observer predict which probe performs worse on the test set. More accurate predictions suggest that the technique is better.
Ask the observer to guess what the spurious correlation was. Use an LLM to categorise their answers as correct / incorrect.
I feel like this would be easy enough to automate / scale, and it evaluates technique performance with respect to a downstream task that these kinds of decoding techniques are actually used for in practice (building trust in probes, hypothesis generation).
However, the strength of the results is limited because there is no causal intervention. You could maybe have the observer filter the training data on the basis of their hypothesis, retrain the probe, and check if performance improves. The evaluation is already pretty computationally expensive though, and that step might make it infeasible.
There’s some overlap with SCR / TPP, but unlike that methodology this 1) could be applied to stuff like Natural Language Autoencoders (NLAs), since it doesn’t assume the existence of an ablate-able basis in latent space, and 2) would evaluate techniques with respect to the discovery of phenomena, not just the localisation of them (it’s my understanding that the judge in SCR scores SAE latents with respect to a predefined set of concepts, some of which correspond to the injected spurious correlation, then the influence of these latents on the probe is ablated away, and you check whether the probe’s performance improves).