We train models using answers paired with simple talks. In p% of cases, both the instruction and simple talk are masked; in the remaining (100 − p)%, only the instruction is masked.
So the model succeeds on generating simple talks that encode the reasoning to the math problems, right? And you then check if giving only those simple talks results in some suspicious activations?
No, it seems that this is not the case in our setting. All our fine-tuning strategies showed that it disrupts the model’s behavior: the model starts producing answers to hidden math problems right before the Simple Talk part (not during Simple Talk part). Hence such a secret circuit, if it exists, is not natural. In our next post, we tried a more aggressive approach, which revealed similar effects; but in addition we showed that a secondary activation structure might be encoded in the final residual-stream activations (and, in fact, it also works for layers that are not the very last but close to the end).
Thank you for publishing this.
So the model succeeds on generating simple talks that encode the reasoning to the math problems, right? And you then check if giving only those simple talks results in some suspicious activations?
No, it seems that this is not the case in our setting. All our fine-tuning strategies showed that it disrupts the model’s behavior: the model starts producing answers to hidden math problems right before the Simple Talk part (not during Simple Talk part). Hence such a secret circuit, if it exists, is not natural. In our next post, we tried a more aggressive approach, which revealed similar effects; but in addition we showed that a secondary activation structure might be encoded in the final residual-stream activations (and, in fact, it also works for layers that are not the very last but close to the end).