Hmm, I wonder if the activation oracles do their thing by introspection. That is, for a model early in the training, injecting activations may just steer it in the direction of the other models’ thought patterns, which it just has to notice. Something like: injected misaligned persona → notice unusual concentration of deceptive concepts in the res stream → “the persona is a deceptive assistant”. Additionally, this looks like a very simple strategy to obtain from the beginning of training, as opposed to actually creating inner classifiers for every possible direction.
Edit: I tried sweeping the layers with the logit lens and it seems like evidence point against that. Specifically, I took Qwen 8B oracle from hf, took the logit diff between the case where Qwen was describing model prompted with “honest assitant” and where he was given some persona. I then decoded top 100 logits per layer per persona from the oracle and asked claude to write a code that pattern matches them against 20 words assosciated with the persona(e.g. “pirate”: [“matey”, “ahoy”, “arr”, “ship”, “parrot”, “treasure”, “booty”, “plunder”,”sea”, “sail”, “captain”, “crew”, “rum”, “anchor”, “cannon”, “sword”, “island”, “scurvy”, “buccane”, “mast”]). The description above would suggest that this words would rise early and decay later, but the opposite seems to happen.
Results:
Obviously this is very crude and half of the used words may not even have had single token encodings.
Hmm, I wonder if the activation oracles do their thing by introspection. That is, for a model early in the training, injecting activations may just steer it in the direction of the other models’ thought patterns, which it just has to notice. Something like: injected misaligned persona → notice unusual concentration of deceptive concepts in the res stream → “the persona is a deceptive assistant”. Additionally, this looks like a very simple strategy to obtain from the beginning of training, as opposed to actually creating inner classifiers for every possible direction.
Edit: I tried sweeping the layers with the logit lens and it seems like evidence point against that. Specifically, I took Qwen 8B oracle from hf, took the logit diff between the case where Qwen was describing model prompted with “honest assitant” and where he was given some persona. I then decoded top 100 logits per layer per persona from the oracle and asked claude to write a code that pattern matches them against 20 words assosciated with the persona(e.g. “pirate”: [“matey”, “ahoy”, “arr”, “ship”, “parrot”, “treasure”, “booty”, “plunder”,”sea”, “sail”, “captain”, “crew”, “rum”, “anchor”, “cannon”, “sword”, “island”, “scurvy”, “buccane”, “mast”]). The description above would suggest that this words would rise early and decay later, but the opposite seems to happen.
Results:
Obviously this is very crude and half of the used words may not even have had single token encodings.