If I understand the “loopies” paper correctly, the main advantage that looping gives you over an “untied” model is that your compute is about 30% faster for the same number of loop-active parameters per token (i.e., counting looped tokens according to their repetition count), allegedly all due to better use of memory bandwidth.
In that paper, they spent their 30% performance improvement into making the model bigger, which unsurprisingly made it smarter. Pretty cool improvement, not world-changing.
In particular, I don’t see anyone claiming that a looped model is significantly better than its equivalent “untied” version in anything but memory utilization. Looping could theoretically gives you a better “inductive bias”, but I personally couldn’t find anyone claiming that this matters for anything.
Of course, more capable models are more capable in engaging in CoT deception, but AFAICT loopies-style looping does not make deception easier than any other linear algebraic efficiency improvement.
If I understand the “loopies” paper correctly, the main advantage that looping gives you over an “untied” model is that your compute is about 30% faster for the same number of loop-active parameters per token (i.e., counting looped tokens according to their repetition count), allegedly all due to better use of memory bandwidth.
In that paper, they spent their 30% performance improvement into making the model bigger, which unsurprisingly made it smarter. Pretty cool improvement, not world-changing.
In particular, I don’t see anyone claiming that a looped model is significantly better than its equivalent “untied” version in anything but memory utilization. Looping could theoretically gives you a better “inductive bias”, but I personally couldn’t find anyone claiming that this matters for anything.
Of course, more capable models are more capable in engaging in CoT deception, but AFAICT loopies-style looping does not make deception easier than any other linear algebraic efficiency improvement.