I don’t agree with the abrupt takeoff assumption. I think a lot of the evidence people use to support that assumption falls apart after further scrutiny. Some examples:
If point 9. refers to the grokking phenomenon, then you should know that grokking does not actually happen very suddenly. It just looks that way in the paper because they use a base-10 log scale on their x-axes. In figure 1, grokking starts to happen roughly 3% of the way through the training process.
The Big-Bench paper suggests the discontinuous improvements seen in language modeling are mostly measurement issues, and more precise quantification of model capabilities suggests smoother arcs of improvement across capabilities.
The “sharp left turn” in human capabilities relative to evolution seems entirely explained by the fact that the inner learning process (an organism’s within-lifetime learning) takes billions of steps for each outer step of evolution, and then dies, and all progress from the inner learner is lost. Human culture somewhat corrected that issue by allowing information to accumulate across generations, and so the greater optimization power used by the inner learning process immediately let the inner learner outstrip the outer optimizer. No need to hypothesize extreme returns on small improvements in generality. See also my comment on whether to expect a sharp left turn in AI training (no, because we won’t spend our compute as stupidly as evolution did).
I don’t agree with the abrupt takeoff assumption. I think a lot of the evidence people use to support that assumption falls apart after further scrutiny. Some examples:
If point 9. refers to the grokking phenomenon, then you should know that grokking does not actually happen very suddenly. It just looks that way in the paper because they use a base-10 log scale on their x-axes. In figure 1, grokking starts to happen roughly 3% of the way through the training process.
The Big-Bench paper suggests the discontinuous improvements seen in language modeling are mostly measurement issues, and more precise quantification of model capabilities suggests smoother arcs of improvement across capabilities.
The “sharp left turn” in human capabilities relative to evolution seems entirely explained by the fact that the inner learning process (an organism’s within-lifetime learning) takes billions of steps for each outer step of evolution, and then dies, and all progress from the inner learner is lost. Human culture somewhat corrected that issue by allowing information to accumulate across generations, and so the greater optimization power used by the inner learning process immediately let the inner learner outstrip the outer optimizer. No need to hypothesize extreme returns on small improvements in generality. See also my comment on whether to expect a sharp left turn in AI training (no, because we won’t spend our compute as stupidly as evolution did).
Thanks! This is very helpful, and yes, I did mean to refer to grokking! Will update the post.