So...I actually think that it technically wasn’t wrong, though the implications that we derived at the time were wrong because reality was more complicated than our simple model.
Roughly, it seems like mental performance is depends on at least two factors: “intelligence” and “knowledge”. It turns out that, at least in some regimes, there’s an exchange rate at which you can make up for mediocre intelligence with massive amounts of knowledge.
My understanding is that this is what’s happening even with the reasoning models. They have a ton of knowledge, including a ton of procedural knowledge about how to solve problems, which is masking the ways in which they’re not very smart.[1]
One way to operationalize how dumb the models are is the number of bits/tokens/inputs/something that are necessary to learn a concept or achieve some performance level on a task. Amortizing over the whole training process / development process, humans are still much more sample efficient learners than foundation models.
Basically, we’ve found a hack where we can get a kind of smart thing to learn a massive amount, which is enough to make it competitive with humans in a bunch of domains. Overall performance is less sensitive to increases in knowledge than to increases in intelligence. This means that we’re traversing the human range of ability, much more slowly than I anticipated we would based on the 2010s LessWrong arguments.
But that doesn’t mean that, for instance, comparatively tiny changes in human brains make the difference between idiots and geniuses.
I’m interested in evidence that bears on this model. Is there evidence that I’m unaware of that’s suggestive that foundation models are smarter than I think, or not relying on knowledge as much as I think?
Is that sentence dumb? Maybe when I’m saying things like that, it should prompt me to refactor my concept of intelligence. Maybe intelligence basically is procedural knowledge of how to solve problems and factoring knowledge and intelligence separately is dumb?
Is that sentence dumb? Maybe when I’m saying things like that, it should prompt me to refactor my concept of intelligence.
I don’t think it’s dumb. But I do think you’re correct that it’s extremely dubious—that we should definitely refactoring the concept of intelligence.
Specifically: There’s default LW-esque frame of some kind of a “core” of intelligence as “general problem solving” apart from any specific bit of knowledge, but I think that—if you manage to turn this belief into a hypothesis rather than a frame—there’s a ton of evidence against this thesis. You could even basically look at the last ~3 years of ML progress as just continuing little bits of evidence against this thesis, month after month after month.
I’m not gonna argue this in a comment, because this is a big thing, but here are some notes around this thesis if you want to tug on the thread.
Comparative psychology finds human infants are characterized by overimmitation relative to Chimpanzees, more than any general problem-solving skill. (That’s a link to a popsci source but there’s a ton of stuff on this.) That is, the skills humans excel at vs. Chimps + Bonobos in experiments are social and allow the quick copying and imitating of others: overimitation, social learning, understanding others as having intentions, etc. The evidence for this is pretty overwhelming, imo.
Ask what Nobel disease seems to say about the general-domain-transfer specificity of human brilliance. Look into scientists with pretty dumb opinions, even when they aren’t getting older. What do people say about the transferability of taste? What does that imply?
How do humans do on even very simple tasks that require reversing heuristics?
Etc etc. Big issue, this is not a complete take, etc. But in general I think LW has an unexamined notion of “intelligence” that feels like it has coherence because of social elaboration, but whose actual predictive validity is very questionable.
All this seems relevant, but there’s still the fact that a human elo at go or chess will improve much more from playing 1000 games (and no more) than an AI playing a 1000 games. That’s suggestive of property learning, or reflection, or conceptualization, or generalization, or something, that the AIs seem to lack, but can compensate for with brute force.
So for the case of our current RL game-playing AIs not learning much from 1000 games—sure, the actual game-playing AIs we have built don’t learn games as efficiently as humans do, in the sense of “from as little data.” But:
Learning from as little data as possible hasn’t actually been a research target, because self-play data is so insanely cheap. So it’s hard to conclude that our current setup for AIs is seriously lacking, because there hasn’t been serious effort to push along this axis.
To point out some areas we could be pushing on, but aren’t: Game-play networks are usually something like ~100x smaller than LLMs, which are themselves ~100-10x smaller than human brains (very approximate numbers). We know from numerous works that data efficiency scales with network size, so even if Adam over matmul is 100% as efficient as human brain matter, we’d still expect our current RL setups to do amazingly poorly with data-efficiency simply because of network size, even leaving aside further issues about lack of hyperparameter search and research effort.
Given this, while this is of course a consideration, it seems far from a conclusive consideration.
Edit: Or more broadly, again—different concepts of “intelligence” will tend to have different areas where they seem to have more predictive use, and different areas they seem to have more epicycles. The areas above are the kind of thing that—if one made them central to one’s notions of intelligence rather than peripheral—you’d probably end up with something different than the LW notion. But again—they certainly do not compel one to do that refactor! It probably wouldn’t make sense to try to do the refactor unless you just keep getting the feeling “this is really awkward / seems off / doesn’t seem to be getting at it some really important stuff” while using the non-refactored notion.
That is, the skills humans excel at vs. Chimps + Bonobos in experiments are social and allow the quick copying and imitating of others: overimitation, social learning, understanding others as having intentions, etc.
Yes, indeed, they copy the actions and play them through their own minds as a method of play, to continue extracting nonobvious concepts. Or at least that is my interpretation. Are you claiming that they are merely copying??
This is very much my gut feeling, too. LLMs have a much greater knowledge base than humans do, and some of them can “think” faster. But humans are still better at many things, including raw problem solving skills. (Though LLM’s problem solving skills have improved a breathtaking amount in the last 12 months since o1-preview shipped. Seriously, folks. The goalpost-moving is giving me vertigo.)
This uneven capabilities profile means that LLMs are still well below the so-called “village idiot” in many important ways, and have already soared past Einstein in others. This averages out to “kinda competent on short time horizons if you don’t squint too hard.”
But even if the difference between “the village idiot” and “smarter than Einstein” involved another AI winter, two major theoretical breakthroughs, and another 10 years, I would still consider that damn close to a vertical curve.
[Epistemic status: unconfident]
So...I actually think that it technically wasn’t wrong, though the implications that we derived at the time were wrong because reality was more complicated than our simple model.
Roughly, it seems like mental performance is depends on at least two factors: “intelligence” and “knowledge”. It turns out that, at least in some regimes, there’s an exchange rate at which you can make up for mediocre intelligence with massive amounts of knowledge.
My understanding is that this is what’s happening even with the reasoning models. They have a ton of knowledge, including a ton of procedural knowledge about how to solve problems, which is masking the ways in which they’re not very smart.[1]
One way to operationalize how dumb the models are is the number of bits/tokens/inputs/something that are necessary to learn a concept or achieve some performance level on a task. Amortizing over the whole training process / development process, humans are still much more sample efficient learners than foundation models.
Basically, we’ve found a hack where we can get a kind of smart thing to learn a massive amount, which is enough to make it competitive with humans in a bunch of domains. Overall performance is less sensitive to increases in knowledge than to increases in intelligence. This means that we’re traversing the human range of ability, much more slowly than I anticipated we would based on the 2010s LessWrong arguments.
But that doesn’t mean that, for instance, comparatively tiny changes in human brains make the difference between idiots and geniuses.
I’m interested in evidence that bears on this model. Is there evidence that I’m unaware of that’s suggestive that foundation models are smarter than I think, or not relying on knowledge as much as I think?
Is that sentence dumb? Maybe when I’m saying things like that, it should prompt me to refactor my concept of intelligence. Maybe intelligence basically is procedural knowledge of how to solve problems and factoring knowledge and intelligence separately is dumb?
I don’t think it’s dumb. But I do think you’re correct that it’s extremely dubious—that we should definitely refactoring the concept of intelligence.
Specifically: There’s default LW-esque frame of some kind of a “core” of intelligence as “general problem solving” apart from any specific bit of knowledge, but I think that—if you manage to turn this belief into a hypothesis rather than a frame—there’s a ton of evidence against this thesis. You could even basically look at the last ~3 years of ML progress as just continuing little bits of evidence against this thesis, month after month after month.
I’m not gonna argue this in a comment, because this is a big thing, but here are some notes around this thesis if you want to tug on the thread.
Comparative psychology finds human infants are characterized by overimmitation relative to Chimpanzees, more than any general problem-solving skill. (That’s a link to a popsci source but there’s a ton of stuff on this.) That is, the skills humans excel at vs. Chimps + Bonobos in experiments are social and allow the quick copying and imitating of others: overimitation, social learning, understanding others as having intentions, etc. The evidence for this is pretty overwhelming, imo.
Take a look at how hard far transfer learning is to get in humans.
Ask what Nobel disease seems to say about the general-domain-transfer specificity of human brilliance. Look into scientists with pretty dumb opinions, even when they aren’t getting older. What do people say about the transferability of taste? What does that imply?
How do humans do on even very simple tasks that require reversing heuristics?
Etc etc. Big issue, this is not a complete take, etc. But in general I think LW has an unexamined notion of “intelligence” that feels like it has coherence because of social elaboration, but whose actual predictive validity is very questionable.
All this seems relevant, but there’s still the fact that a human elo at go or chess will improve much more from playing 1000 games (and no more) than an AI playing a 1000 games. That’s suggestive of property learning, or reflection, or conceptualization, or generalization, or something, that the AIs seem to lack, but can compensate for with brute force.
So for the case of our current RL game-playing AIs not learning much from 1000 games—sure, the actual game-playing AIs we have built don’t learn games as efficiently as humans do, in the sense of “from as little data.” But:
Learning from as little data as possible hasn’t actually been a research target, because self-play data is so insanely cheap. So it’s hard to conclude that our current setup for AIs is seriously lacking, because there hasn’t been serious effort to push along this axis.
To point out some areas we could be pushing on, but aren’t: Game-play networks are usually something like ~100x smaller than LLMs, which are themselves ~100-10x smaller than human brains (very approximate numbers). We know from numerous works that data efficiency scales with network size, so even if Adam over matmul is 100% as efficient as human brain matter, we’d still expect our current RL setups to do amazingly poorly with data-efficiency simply because of network size, even leaving aside further issues about lack of hyperparameter search and research effort.
Given this, while this is of course a consideration, it seems far from a conclusive consideration.
Edit: Or more broadly, again—different concepts of “intelligence” will tend to have different areas where they seem to have more predictive use, and different areas they seem to have more epicycles. The areas above are the kind of thing that—if one made them central to one’s notions of intelligence rather than peripheral—you’d probably end up with something different than the LW notion. But again—they certainly do not compel one to do that refactor! It probably wouldn’t make sense to try to do the refactor unless you just keep getting the feeling “this is really awkward / seems off / doesn’t seem to be getting at it some really important stuff” while using the non-refactored notion.
and whose predictive validity in humans doesn’t transfer well across cognitive architectures. e.g. reverse digit span.
Yes, indeed, they copy the actions and play them through their own minds as a method of play, to continue extracting nonobvious concepts. Or at least that is my interpretation. Are you claiming that they are merely copying??
This is very much my gut feeling, too. LLMs have a much greater knowledge base than humans do, and some of them can “think” faster. But humans are still better at many things, including raw problem solving skills. (Though LLM’s problem solving skills have improved a breathtaking amount in the last 12 months since o1-preview shipped. Seriously, folks. The goalpost-moving is giving me vertigo.)
This uneven capabilities profile means that LLMs are still well below the so-called “village idiot” in many important ways, and have already soared past Einstein in others. This averages out to “kinda competent on short time horizons if you don’t squint too hard.”
But even if the difference between “the village idiot” and “smarter than Einstein” involved another AI winter, two major theoretical breakthroughs, and another 10 years, I would still consider that damn close to a vertical curve.