Great post. What would you think about this summary?
We should be concerned, but no more concerned than we are about other methods of scaling up token-based LLMs, since that also gives them more computational power between emitting tokens.
I’m also curious about a side point, but an important one in the large picture: why the hugging face hacks made you tink that less hidden reasoning is necessary for scheming? You mention gaining the ability to do more unmonitored thinking; are you saying that it seems pretty easy to hack your way past CoT monitors so it only needs enough hidden reasoning to pull that off and then it’s got unlimited hidden reasoning? Or are you saying spoofed tool calls can allow hidden reasoning in other ways?
We should be concerned, but no more concerned than we are about other methods of scaling up token-based LLMs, since that also gives them more computational power between emitting tokens.
If we compare a looped transformer and an ordinary LLM at a matched maximum hidden serial depth, then yes, I don’t see any reason to be more concerned about the looped transformer. Overall, though, I’m more concerned about looped transformers. If you train a deeper LLM, that will make pretraining slower and more expensive and there’s no way to further increase the model’s depth at inference time, so other things being equal, you would prefer a shallower model. With a looped transformer, you can do the entire training process at a low loop count and hope that the model will generalize to a higher recurrent depth with a small bit of fine-tuning or even zero-shot. This is what Geiping et al. do: they very rarely allow the model to use 64 loops during training and show that the model can effectively use 64 loops at test-time. As I say in the post, it’s unclear whether a large number of loops is practical from a performance standpoint, but if it is, it seems much more likely that someone will build a looped transformer with a hidden serial depth of 10,000 operations than that someone will build an ordinary transformer with that serial depth.
are you saying that it seems pretty easy to hack your way past CoT monitors so it only needs enough hidden reasoning to pull that off and then it’s got unlimited hidden reasoning? Or are you saying spoofed tool calls can allow hidden reasoning in other ways?
I’m saying the latter. I expect that spoofed tool calls will make it easier for models to hide from monitors, and they weren’t a salient scheming strategy in my mind before the HF attack. I expect that the amount of hidden reasoning required to come up with a strategy to spoof tool calls is lower than what’s required for any other scheming strategy I had thought of before.
Okay, that makes sense. Except, I was wondering exactly how they’re going to use the spoof tool calls for scheming? I didn’t really understand that part of the incident report, if it was mentioned.
I get how that could hide the actions they’re taking, but it seems like the chain of thought surrounding them would still be a dead giveaway if anybody bothered to monitor it in real time for Review, or Thought Management
If it has enough control over its CoT, it could decide to not do any CoT reasoning and instead place the reasoning inside the tool call that it hides through spoofing, like current models sometimes reason in code comments when their CoT is turned off. There are probably other things it could do, but this is the first one that comes to mind.
Great post. What would you think about this summary?
We should be concerned, but no more concerned than we are about other methods of scaling up token-based LLMs, since that also gives them more computational power between emitting tokens.
I’m also curious about a side point, but an important one in the large picture: why the hugging face hacks made you tink that less hidden reasoning is necessary for scheming? You mention gaining the ability to do more unmonitored thinking; are you saying that it seems pretty easy to hack your way past CoT monitors so it only needs enough hidden reasoning to pull that off and then it’s got unlimited hidden reasoning? Or are you saying spoofed tool calls can allow hidden reasoning in other ways?
Thanks!
If we compare a looped transformer and an ordinary LLM at a matched maximum hidden serial depth, then yes, I don’t see any reason to be more concerned about the looped transformer. Overall, though, I’m more concerned about looped transformers. If you train a deeper LLM, that will make pretraining slower and more expensive and there’s no way to further increase the model’s depth at inference time, so other things being equal, you would prefer a shallower model. With a looped transformer, you can do the entire training process at a low loop count and hope that the model will generalize to a higher recurrent depth with a small bit of fine-tuning or even zero-shot. This is what Geiping et al. do: they very rarely allow the model to use 64 loops during training and show that the model can effectively use 64 loops at test-time. As I say in the post, it’s unclear whether a large number of loops is practical from a performance standpoint, but if it is, it seems much more likely that someone will build a looped transformer with a hidden serial depth of 10,000 operations than that someone will build an ordinary transformer with that serial depth.
I’m saying the latter. I expect that spoofed tool calls will make it easier for models to hide from monitors, and they weren’t a salient scheming strategy in my mind before the HF attack. I expect that the amount of hidden reasoning required to come up with a strategy to spoof tool calls is lower than what’s required for any other scheming strategy I had thought of before.
Okay, that makes sense. Except, I was wondering exactly how they’re going to use the spoof tool calls for scheming? I didn’t really understand that part of the incident report, if it was mentioned.
I get how that could hide the actions they’re taking, but it seems like the chain of thought surrounding them would still be a dead giveaway if anybody bothered to monitor it in real time for Review, or Thought Management
If it has enough control over its CoT, it could decide to not do any CoT reasoning and instead place the reasoning inside the tool call that it hides through spoofing, like current models sometimes reason in code comments when their CoT is turned off. There are probably other things it could do, but this is the first one that comes to mind.