It’s a central example of a research task that requires insight and developing new, deep, technical concepts, not just patternmatching and reasoning by analogy to similar-seeming past problems.
(I wouldn’t use these words probably, but it’s a fine gesture in the direction.)
But everyone seems to be eliding this distinction between fluid intelligence and crystalized intelligence!
[Acknowledging that you acknowledged the ontology skew] Meh. I don’t think “fluid” vs. “crystallized” is all that important / clear / useful a distinction. It kinda sorta gets at the things, and other people bring it up so I use it. IDK if other people elide that distinction. I’d talk more about general intelligence, though that packs in additional stuff; I’m using something spiritually like Yudkowsky’s definition; something like “cross-domain optimization power divided by inputs”. Also gestured here: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#AGI
Some people don’t seem to notice the difference at all, and think that the crystallized intelligence of the AIs is the same kind of thing as the deep fluid intelligence.
IDK if this is true.
This is a kind of obvious point, but everyone around me seems to be operating on a strategic model that apparently ignores it!
No, I don’t think there’s one specific point I think everyone’s missing, or everyone with short timelines is missing. If there were, I suppose it would be more like “notice that you’re deferring way more than you’re noticing, and that others are too, and that this is creating very bad epistemic tidal forces” or something, but that’s kinda hard to discuss productively. I keep repeating that I don’t understand how you / others got to be so confident in short timelines in part so that it’s clear that IDK what you’re missing even on my perspective. I try to ask people to lay out their reasoning more in order to bring out background assumptions that I can disagree with better, which background assumptions support the [past x years of LLM observations] --> [confident short timelines] update. But that doesn’t work that well / is frustrating.
For example, I think that besides the “crystallizes int could make fluid int” thing, you’re also saying “we have fluid int or are close to it probably”. I’m asking about that and wanting to argue against that (that is, argue against the reasoning I’ve heard so far as not supporting the stated confidence). IDK how to phrase that as a positive statement for a position summary, because I would normally view it as weird / the wrong order of operations to start negating arguments (whatever your specific arguments are) without hearing them first, but I suppose I could preliminarily say “I’m guessing that you made a wrong update from observations to timelines based on a misconstrual of what matters about intelligence and what is the causal structure between cognitive faculties and cognitive performances.”, as described here
https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=QBca6vhdeKkjyNLKa
and here
https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=DDaz5zcETcuyuy5Xx . But again, that feels awkward to guess about given that I don’t understand your reasons for the update.
you’re also saying “we have fluid int or are close to it probably”.
I think I’m saying “crystalized intelligence can, to a large extent, substitute for fluid intelligence”. This is true to an extent, of humans, but it’s much more true of AIs, because they can have so much more crystalized intelligence than any human could hope to attain.
This is relevant to modeling if LLM agents will transform the world and to modeling if LLM agents will rapidly give way to something that’s much more capable.
In particular, I (unconfidently) dispute that developing an AI with fluid intelligence is a research project that is itself heavily and crucially loaded on fluid intelligence.[1]
On my view, it is pretty likely that huge amounts of superhuman crystalized intelligence can find fluid-intelligence-emulating mechanisms (possibly with a necessary ingredient of a relatively small amount of genius fluid intelligence that even huge amounts of crystalized intelligence can’t substitute for, or possibly even without that input).
In that sense, we’re “close to” solving fluid intelligence, even if there’s decades of subjective research an iteration time between here and there.
I do additionally suspect that mechanisms to implement fluid intelligence are just not that hard to invent and/or scale to, starting from the AI tech of 2026. Like, it seems somewhat likely to me that that various dumb ideas would just totally work, or that doing the same stuff we’ve already been doing, but more so will totally work. However, I’m much less confident about this point, and perhaps you can teach me some things that would quickly cause me to change my mind.
I want to check if this comment is clarifying, or if it feel like me repeating things that I’ve already said.
Is there a way you could restate this in terms of one or more propositions that are fairly load-bearing for your confident short timelines? Is it this?
it is pretty likely that huge amounts of superhuman crystalized intelligence can find fluid-intelligence-emulating mechanisms
I’m struggling to get clear on the logical structure of your beliefs. Cf. your comments
Like, it seems like you can maybe get a strategically superhuman AI by relying on a lattice of more-or-less specialized superhuman skills (including superhuman engineering, and superhuman persuasion, and superhuman corporate strategy, and so on), without having much fluid intelligence.
To be clear, it also seems possible to me that we will make superhuman AI agents that don’t have this fluid intelligence special sauce. Those AIs will be adequate to automate almost all human labor, because almost all of human labor is more-or-less routine application of crystalized knowledge. We’ll be living in a radical new world of ~full automation, except for a small number of geniuses who are adding critical insight steps to the new cyborg-process of doing science.[1]
But I will be surprised if we hang out in that regime for very long, before the combined might of humanity’s geniuses augmented their armies of superhumanly capable routine engineers, and enormous computer infrastructure to do massive experiments, can’t hit on a mechanism that replicates the human fluid intelligence special sauce.[2]
and
I’m putting forward a disjunction:
Fluid intelligence isn’t necessary for Strategically Superhuman AI.or
LLM based agents will develop fluid intelligence on the default technological trajectory, via the application of not-very-clever ideas.or
There’s about one or two “breakthrough” ideas missing, that when combined with the existing LLM-agent techniques, will make LLM-agents that can do the fluid intelligence thing (or a substitute for the fluid intelligence thing). Having armies of LLM-agents that can automate engineering and experimentation seems like it should accelerate the discovery of those one or two breakthroughs.
Those last two legs of the disjunction are assuming that there are not many pieces left before fluid intelligence is solved, but not making much of a claim about how many pieces we already have. Like, depending on what one means by “pieces”, maybe we have 0 out of 1 (and we’re likely to get that one in the next five years), or maybe we have 95 out of 100 (and we’re likely to get the last five in the next five years).
and
Like, if I thought that we were conceptually / technically far from AIs that can automate the process of scientific discovery, I would much less expect a FOOM in the next 10 years (though we would still have an emergency, because automating science isn’t a necessary capability for destabilizing or ending the world via any of a number of different pathways).
And
I think I’m saying “crystalized intelligence can, to a large extent, substitute for fluid intelligence”.
Can you please clarify what are main big thingies that you expect to happen fairly confidently within 10 years? Can you clarify what drives your confident beliefs in those thingies happening? E.g. are you confident in some X happening due to being confident that we have fluid intelligence already or are close, or is that not the case?
Maybe you could make up words for the main scenarios and main capabilities / faculties you’re thinking of?
To mods (@Raemon): this would be an example of a place where a “have an LLM summarize the exchanges in this thread between these two users, with links / quotes” button could be helpful.
(Also, it seems that the option to temporarily switch to LW editor was disabled, so I cannot tag people?)
I’ve been reading this thread and feeling motivated to at least make it easier to export the conversation to clipboard so you can more easily paste into LLM (in my case so I can argue with the AI about what obvious objections you’d have to what I’d say next).
I’m a bit confused about what’s going on with the markdown editor. I’ve asked other mods.
Thanks.
(I wouldn’t use these words probably, but it’s a fine gesture in the direction.)
[Acknowledging that you acknowledged the ontology skew] Meh. I don’t think “fluid” vs. “crystallized” is all that important / clear / useful a distinction. It kinda sorta gets at the things, and other people bring it up so I use it. IDK if other people elide that distinction. I’d talk more about general intelligence, though that packs in additional stuff; I’m using something spiritually like Yudkowsky’s definition; something like “cross-domain optimization power divided by inputs”. Also gestured here: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#AGI
IDK if this is true.
No, I don’t think there’s one specific point I think everyone’s missing, or everyone with short timelines is missing. If there were, I suppose it would be more like “notice that you’re deferring way more than you’re noticing, and that others are too, and that this is creating very bad epistemic tidal forces” or something, but that’s kinda hard to discuss productively. I keep repeating that I don’t understand how you / others got to be so confident in short timelines in part so that it’s clear that IDK what you’re missing even on my perspective. I try to ask people to lay out their reasoning more in order to bring out background assumptions that I can disagree with better, which background assumptions support the [past x years of LLM observations] --> [confident short timelines] update. But that doesn’t work that well / is frustrating.
For example, I think that besides the “crystallizes int could make fluid int” thing, you’re also saying “we have fluid int or are close to it probably”. I’m asking about that and wanting to argue against that (that is, argue against the reasoning I’ve heard so far as not supporting the stated confidence). IDK how to phrase that as a positive statement for a position summary, because I would normally view it as weird / the wrong order of operations to start negating arguments (whatever your specific arguments are) without hearing them first, but I suppose I could preliminarily say “I’m guessing that you made a wrong update from observations to timelines based on a misconstrual of what matters about intelligence and what is the causal structure between cognitive faculties and cognitive performances.”, as described here https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=QBca6vhdeKkjyNLKa and here https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=DDaz5zcETcuyuy5Xx . But again, that feels awkward to guess about given that I don’t understand your reasons for the update.
I think I’m saying “crystalized intelligence can, to a large extent, substitute for fluid intelligence”. This is true to an extent, of humans, but it’s much more true of AIs, because they can have so much more crystalized intelligence than any human could hope to attain.
This is relevant to modeling if LLM agents will transform the world and to modeling if LLM agents will rapidly give way to something that’s much more capable.
In particular, I (unconfidently) dispute that developing an AI with fluid intelligence is a research project that is itself heavily and crucially loaded on fluid intelligence.[1]
On my view, it is pretty likely that huge amounts of superhuman crystalized intelligence can find fluid-intelligence-emulating mechanisms (possibly with a necessary ingredient of a relatively small amount of genius fluid intelligence that even huge amounts of crystalized intelligence can’t substitute for, or possibly even without that input).
In that sense, we’re “close to” solving fluid intelligence, even if there’s decades of subjective research an iteration time between here and there.
I do additionally suspect that mechanisms to implement fluid intelligence are just not that hard to invent and/or scale to, starting from the AI tech of 2026. Like, it seems somewhat likely to me that that various dumb ideas would just totally work, or that doing the same stuff we’ve already been doing, but more so will totally work. However, I’m much less confident about this point, and perhaps you can teach me some things that would quickly cause me to change my mind.
I want to check if this comment is clarifying, or if it feel like me repeating things that I’ve already said.
Though you flagged that those aren’t the words that you would use, so
Is there a way you could restate this in terms of one or more propositions that are fairly load-bearing for your confident short timelines? Is it this?
I’m struggling to get clear on the logical structure of your beliefs. Cf. your comments
and
and
And
Can you please clarify what are main big thingies that you expect to happen fairly confidently within 10 years? Can you clarify what drives your confident beliefs in those thingies happening? E.g. are you confident in some X happening due to being confident that we have fluid intelligence already or are close, or is that not the case?
Maybe you could make up words for the main scenarios and main capabilities / faculties you’re thinking of?
To mods (@Raemon): this would be an example of a place where a “have an LLM summarize the exchanges in this thread between these two users, with links / quotes” button could be helpful.
(Also, it seems that the option to temporarily switch to LW editor was disabled, so I cannot tag people?)
I’ve been reading this thread and feeling motivated to at least make it easier to export the conversation to clipboard so you can more easily paste into LLM (in my case so I can argue with the AI about what obvious objections you’d have to what I’d say next).
I’m a bit confused about what’s going on with the markdown editor. I’ve asked other mods.
(actually I asked for Gemini 3.1 pro preview for a summary of a copypasted version and it wasn’t good)