I agree, but I feel like there’s an even simpler and more obvious explanation for what’s going on: Just look at the training data!
--When a modern AI is first created, it’s a random jumble of neurons just like a human fetus. --Then it plays the “predict the next token of internet text” game ten trillion times. --Then it plays the “Here’s a puzzle to solve / bug to fix / short-term task to complete / question to answer” game like ten million times. --Then, bam, there it is, in front of you, being asked to rebuild your video game from scratch or do novel research or whatever hard task you are asking it to do. Or play Pokemon for that matter. These tasks are quite different from anything it’s seen before: much longer, much harder. To solve them it’ll need to learn and adapt over the course of days. But it’s literally never had to learn and adapt over the course of days before; the longest tasks it ever saw in training were shorter than that, and weren’t very diverse either.
I think I like half agree, but also think your argument implies things which are false.
In particular, I don’t think the reason why humans are much better at much longer tasks is mostly because they’ve previously done longer tasks. I think humans trained on only a variety of school projects that took around 8 hours (but were hard annd variable + diverse within that period), would be pretty likely to suceed at year long projects without further feedback / training on longer tasks. They would struggle at the start but would probably figure it out and eventually suceed with only the feedback from the environment. Minimally, humans only need a tiny number of examples of long tasks.
I disagree. Humans who do research projects well have usually had experience doing at least a few research projects of similar length before. Indeed when I was a grad student teaching philosophy courses, it seemed to me that part of the problem was that the college students I was teaching had only ever written short 3-paragraph-style essays before, and never done serious lengthy writing projects that required serious reading and thinking.
It’s true that humans need only a few examples of the thing to get decently good at it, but that’s just the generic human data-efficiency at work rather than specific to long-horizon tasks, I think.
Sure, but the training data thing is sort of tied up with fluid intelligence? Like a method you have available as a human is learning on the task itself and AIs also have this option, but due to garbage sample efficiency it wouldn’t work.
I also still think humans can often generalize to doing tasks of larger scope without prior experience by learning/adapting from this exact experience in a way that AIs don’t.
Like a method you have available as a human is learning on the task itself and AIs also have this option, but due to garbage sample efficiency it wouldn’t work.
Sample efficiency for RLVR is plausibly good enough if relevant tasks and RL environments can be automatically formulated, which is currently out of reach but it’s unclear for how long it stays this way. And for in-context learning, sample efficiency could get higher if it worked well enough (as learned learning, it’s in principle not constrained by what hardcoded learning can do), it just doesn’t work well with pretraining.
I think I was responding to an earlier version of your comment—now that it says “I half agree” then I think we are mostly on the same page. I also agree that long-horizon fluid intelligence / sample-efficient online learning seems to be a real thing that differs between humans and AIs.
There isn’t enough samples in “human training data” to train them for agentic long horizon tasks either. There is, however, enough of those samples in the human evolutionary history. Which is how humans have an instinctual grasp of how to accomplish long term goals—which forms a foundation that any further training then builds upon.
LLMs don’t get that “for free”, and they struggle to learn it from random pre-training text.
Normally, the AI answer to “no hardwired priors” is “reconstruct them the hard way from the vast dataset”. But there’s no good “in-domain dataset of long term agentic behavior” for LLMs to be able to do it well. So they get stuck with dregs of long term agency that the mix of pre-training, SFT and RLVR left them with.
I agree, but I feel like there’s an even simpler and more obvious explanation for what’s going on: Just look at the training data!
--When a modern AI is first created, it’s a random jumble of neurons just like a human fetus.
--Then it plays the “predict the next token of internet text” game ten trillion times.
--Then it plays the “Here’s a puzzle to solve / bug to fix / short-term task to complete / question to answer” game like ten million times.
--Then, bam, there it is, in front of you, being asked to rebuild your video game from scratch or do novel research or whatever hard task you are asking it to do. Or play Pokemon for that matter. These tasks are quite different from anything it’s seen before: much longer, much harder. To solve them it’ll need to learn and adapt over the course of days. But it’s literally never had to learn and adapt over the course of days before; the longest tasks it ever saw in training were shorter than that, and weren’t very diverse either.
I think I like half agree, but also think your argument implies things which are false.
In particular, I don’t think the reason why humans are much better at much longer tasks is mostly because they’ve previously done longer tasks. I think humans trained on only a variety of school projects that took around 8 hours (but were hard annd variable + diverse within that period), would be pretty likely to suceed at year long projects without further feedback / training on longer tasks. They would struggle at the start but would probably figure it out and eventually suceed with only the feedback from the environment. Minimally, humans only need a tiny number of examples of long tasks.
I disagree. Humans who do research projects well have usually had experience doing at least a few research projects of similar length before. Indeed when I was a grad student teaching philosophy courses, it seemed to me that part of the problem was that the college students I was teaching had only ever written short 3-paragraph-style essays before, and never done serious lengthy writing projects that required serious reading and thinking.
It’s true that humans need only a few examples of the thing to get decently good at it, but that’s just the generic human data-efficiency at work rather than specific to long-horizon tasks, I think.
Sure, but the training data thing is sort of tied up with fluid intelligence? Like a method you have available as a human is learning on the task itself and AIs also have this option, but due to garbage sample efficiency it wouldn’t work.
I also still think humans can often generalize to doing tasks of larger scope without prior experience by learning/adapting from this exact experience in a way that AIs don’t.
Sample efficiency for RLVR is plausibly good enough if relevant tasks and RL environments can be automatically formulated, which is currently out of reach but it’s unclear for how long it stays this way. And for in-context learning, sample efficiency could get higher if it worked well enough (as learned learning, it’s in principle not constrained by what hardcoded learning can do), it just doesn’t work well with pretraining.
I think I was responding to an earlier version of your comment—now that it says “I half agree” then I think we are mostly on the same page. I also agree that long-horizon fluid intelligence / sample-efficient online learning seems to be a real thing that differs between humans and AIs.
Hardwired priors. Humans get them, LLMs don’t.
There isn’t enough samples in “human training data” to train them for agentic long horizon tasks either. There is, however, enough of those samples in the human evolutionary history. Which is how humans have an instinctual grasp of how to accomplish long term goals—which forms a foundation that any further training then builds upon.
LLMs don’t get that “for free”, and they struggle to learn it from random pre-training text.
Normally, the AI answer to “no hardwired priors” is “reconstruct them the hard way from the vast dataset”. But there’s no good “in-domain dataset of long term agentic behavior” for LLMs to be able to do it well. So they get stuck with dregs of long term agency that the mix of pre-training, SFT and RLVR left them with.