No, it seems that this is not the case in our setting. All our fine-tuning strategies showed that it disrupts the model’s behavior: the model starts producing answers to hidden math problems right before the Simple Talk part (not during Simple Talk part). Hence such a secret circuit, if it exists, is not natural. In our next post, we tried a more aggressive approach, which revealed similar effects; but in addition we showed that a secondary activation structure might be encoded in the final residual-stream activations (and, in fact, it also works for layers that are not the very last but close to the end).
No, it seems that this is not the case in our setting. All our fine-tuning strategies showed that it disrupts the model’s behavior: the model starts producing answers to hidden math problems right before the Simple Talk part (not during Simple Talk part). Hence such a secret circuit, if it exists, is not natural. In our next post, we tried a more aggressive approach, which revealed similar effects; but in addition we showed that a secondary activation structure might be encoded in the final residual-stream activations (and, in fact, it also works for layers that are not the very last but close to the end).