Some further thoughts:
If you look at some of the CoT from models where we can see it, e.g. Deepseek-V3, the obvious conclusion is that these are not yet optimal paths towards solutions.
Look at the token cost for o3 solving ARC-AGI. It was basically writing multiple novels worth of CoT to solve fairly simple puzzles. A human ‘thinking out loud’ and giving very detailed CoT would manage to solve those puzzles in well less than 0.1% that many tokens.
My take on this is that this means there is lots of room for improvement. The techniques haven’t been “maxed out” yet. We are on the beginning of the steep part of the S-curve for this specific tech.
I had a similar reflection yesterday regarding these inference-time techniques (post-training, unhobbling, whatever you want to call it) being in the very early days. Would it be too much of a stretch to draw parallels here between how such unhobbling methods lead to an explosion of human capabilities over the past ~10000 years? The human DNA has undergone roughly the same number of ‘gradient updates’ (evolutionary cycles) as our predecessors from a few millenia ago. I see it as having an equivalent amount of training compute. Yet through an efficient use of tools, language, writing, coordination and similar, we have completely outdone what our ancestors were able to do.
There is a difference in that for us, these abilities arose naturally through evolution. We are now manually engineering them into AI systems. I would not be surprised to see a real capability explosion soon (much faster than what we are observing now) - not because of the continued scaling up of pre-training, but because of these post-training enhancements.
Going to link a previous comment of mine from a similar discussion: https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=rNycG6oyuMDmnBaih This comment addresses the point of: what happens after the AI gets strong enough to substantially accelerate algorithmic improvement (e.g. above 2x)? You can no longer assume that the trends will hold.
Some further thoughts: If you look at some of the CoT from models where we can see it, e.g. Deepseek-V3, the obvious conclusion is that these are not yet optimal paths towards solutions. Look at the token cost for o3 solving ARC-AGI. It was basically writing multiple novels worth of CoT to solve fairly simple puzzles. A human ‘thinking out loud’ and giving very detailed CoT would manage to solve those puzzles in well less than 0.1% that many tokens.
My take on this is that this means there is lots of room for improvement. The techniques haven’t been “maxed out” yet. We are on the beginning of the steep part of the S-curve for this specific tech.
I had a similar reflection yesterday regarding these inference-time techniques (post-training, unhobbling, whatever you want to call it) being in the very early days. Would it be too much of a stretch to draw parallels here between how such unhobbling methods lead to an explosion of human capabilities over the past ~10000 years? The human DNA has undergone roughly the same number of ‘gradient updates’ (evolutionary cycles) as our predecessors from a few millenia ago. I see it as having an equivalent amount of training compute. Yet through an efficient use of tools, language, writing, coordination and similar, we have completely outdone what our ancestors were able to do.
There is a difference in that for us, these abilities arose naturally through evolution. We are now manually engineering them into AI systems. I would not be surprised to see a real capability explosion soon (much faster than what we are observing now) - not because of the continued scaling up of pre-training, but because of these post-training enhancements.
Here’s what I mean about being on the steep part of the S-curve: https://x.com/OfficialLoganK/status/1885374062098018319