Broadly speaking, autonomous learning doesn’t seem particularly distinguished relative to supervised learning unless you have data limitations.
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
Option A: You need to spend the entire week reading AI literature. I choose what you read, and in what order, using a random number generator and selecting out of every AI paper / textbook ever written. While reading, you are forced to dwell for exactly one second—no more, no less—on each word of the text, before moving on to the next word.
Option B: You can spend your week however you want. Follow the threads that seem promising, sit and think for a while, go back and re-read passages that are confusing, etc.
It seems extremely obvious to me that you’d make more progress under Option B than Option A—like, massively, qualitatively more progress. Do you not share that intuition? (See also Section 1.1 here.)
(Note: this comment is rambly and repetitive, but I decided not to spend time cleaning it up)
It sounds like you believe something like:
“There are autonomous learning style approaches which are considerably better than the efficiency on next token prediction.”
And more broadly, you’re making a claim like ‘current learning efficiency is very low’.
I agree—brains imply that it’s possible to learn vastly more efficiently than deep nets, and my guess would be that performance can be far, far better than brains.
Suppose we instantly went from ‘current status quo’ to ‘AI systems learn like humans learn and with the same efficiency, but with vastly larger memories than humans (current LLMs seem to have vastly better memory at least for facts and technical minutia), and vastly longer lifespans than humans (if you think token corresponds to 1 second, then 10 trillion tokens is 317098 years!)’. Then, we certainly get an extremely hard FOOM if anyone runs this training!
But this hypothetical just isn’t what I expect.
Currently, SOTA deep learning is deeply inefficient in a bunch of different ways. Failing to do open ended autonomous learning to advance a field and then distilling these insights down to allow for future progress is probably one such failure, but I don’t think it seem particularly special. Nor do I see a particular reason to expect that advances in open ended flexible autonomous learning will be considerably more jumpy than advances in other domains.
Right now, both supervised next token prediction and fully flexible autonomous learning are far less efficient than theoretical limits and worse than brains. But currently next token prediction is more efficient than fully flexible autonomous learning (as the main way to train your AI, next token prediction + some other stuff is often used).
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
In this hypothetical, I obviously would pick option B.
But suppose instead that we asked “How would you try to get current AIs (without technical advances) to most efficiently come up with new good experiements to try?”
Then, my guess is that most of the flops go toward next token prediction or a similar objective on a huge corpus of data.
You’d then do some RL(HF) and/or amplification to try and improve further, but this would be a small fraction of overall training.
As AIs get smarter, clever techniques to improve their capabilities futher via ‘self improvement’ will continue to work better and better, but I don’t think this clearly will end up being where you spend most of the flops (it’s certainly possible, but I don’t see a particular reason to expect this—it could go either way).
I agree that ‘RL on thoughts’ might prove important, but we already have shitty versions today. Current SOTA is probably like ‘process based feedback’ + ‘some outcomes’ + ‘amplification’ + ‘etc’. Noteably this is how humans do things: we reflect on which cognitive strategies and thoughts were good and then try to do more of that. ‘thoughts’ isn’t really doing that much work here—this is just standard stuff. I expect continued progress on these techniques and that techiques will work better and better for smarter models. But I don’t expect massive sharp left turn advancements for the reasons given above.
Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.
You’re thinking about inference, and I’m thinking about learning. When I spend my week trying to come up with the project, I’m permanently smarter at the end of the week than I was at the beginning. It’s a weights-versus-context-window thing. I think weight-learning can do things that context-window-“learning” can’t. In my mind, this belief is vaguely related to my belief that there is no possible combination of sensory inputs that will give a human a deep understanding of chemistry from scratch in 10 minutes. (And having lots of clones of that human working together doesn’t help.)
Suppose I ask you to spend a week trying to come up with a new good experiment to try in AI. I give you two options.
Option A: You need to spend the entire week reading AI literature. I choose what you read, and in what order, using a random number generator and selecting out of every AI paper / textbook ever written. While reading, you are forced to dwell for exactly one second—no more, no less—on each word of the text, before moving on to the next word.
Option B: You can spend your week however you want. Follow the threads that seem promising, sit and think for a while, go back and re-read passages that are confusing, etc.
It seems extremely obvious to me that you’d make more progress under Option B than Option A—like, massively, qualitatively more progress. Do you not share that intuition? (See also Section 1.1 here.)
(Note: this comment is rambly and repetitive, but I decided not to spend time cleaning it up)
It sounds like you believe something like: “There are autonomous learning style approaches which are considerably better than the efficiency on next token prediction.”
And more broadly, you’re making a claim like ‘current learning efficiency is very low’.
I agree—brains imply that it’s possible to learn vastly more efficiently than deep nets, and my guess would be that performance can be far, far better than brains.
Suppose we instantly went from ‘current status quo’ to ‘AI systems learn like humans learn and with the same efficiency, but with vastly larger memories than humans (current LLMs seem to have vastly better memory at least for facts and technical minutia), and vastly longer lifespans than humans (if you think token corresponds to 1 second, then 10 trillion tokens is 317098 years!)’. Then, we certainly get an extremely hard FOOM if anyone runs this training!
But this hypothetical just isn’t what I expect.
Currently, SOTA deep learning is deeply inefficient in a bunch of different ways. Failing to do open ended autonomous learning to advance a field and then distilling these insights down to allow for future progress is probably one such failure, but I don’t think it seem particularly special. Nor do I see a particular reason to expect that advances in open ended flexible autonomous learning will be considerably more jumpy than advances in other domains.
Right now, both supervised next token prediction and fully flexible autonomous learning are far less efficient than theoretical limits and worse than brains. But currently next token prediction is more efficient than fully flexible autonomous learning (as the main way to train your AI, next token prediction + some other stuff is often used).
In this hypothetical, I obviously would pick option B.
But suppose instead that we asked “How would you try to get current AIs (without technical advances) to most efficiently come up with new good experiements to try?”
Then, my guess is that most of the flops go toward next token prediction or a similar objective on a huge corpus of data.
You’d then do some RL(HF) and/or amplification to try and improve further, but this would be a small fraction of overall training.
As AIs get smarter, clever techniques to improve their capabilities futher via ‘self improvement’ will continue to work better and better, but I don’t think this clearly will end up being where you spend most of the flops (it’s certainly possible, but I don’t see a particular reason to expect this—it could go either way).
I agree that ‘RL on thoughts’ might prove important, but we already have shitty versions today. Current SOTA is probably like ‘process based feedback’ + ‘some outcomes’ + ‘amplification’ + ‘etc’. Noteably this is how humans do things: we reflect on which cognitive strategies and thoughts were good and then try to do more of that. ‘thoughts’ isn’t really doing that much work here—this is just standard stuff. I expect continued progress on these techniques and that techiques will work better and better for smarter models. But I don’t expect massive sharp left turn advancements for the reasons given above.
Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.
It seems like retrieval + chain of thought mostly just solves this already
You’re thinking about inference, and I’m thinking about learning. When I spend my week trying to come up with the project, I’m permanently smarter at the end of the week than I was at the beginning. It’s a weights-versus-context-window thing. I think weight-learning can do things that context-window-“learning” can’t. In my mind, this belief is vaguely related to my belief that there is no possible combination of sensory inputs that will give a human a deep understanding of chemistry from scratch in 10 minutes. (And having lots of clones of that human working together doesn’t help.)
Distilling inference based approaches into learning is usually reasonably straightforward. I think this also applies in this case.
This doesn’t necessarily apply to ‘learning how to learn’.
(That said, I’m less sold that retrieval + chain of thought ‘mostly solves autonmomous learning’)