I’m wondering if it’s a necessary component to an Eliezer-like view on general intelligence and hard takeoff that present-day machine learning methods won’t get us to general intelligence.
I know Eliezer expects there to be algorithmic secret sauce. But another feature of his view seems to be that the “progress space (design-differences-to-usefulness-of-output)” is densely packed at the stage where you reach general intelligence. That’s what his human examples (von Neumann, Jobs) are always driving at: small differences in brain designs generate vast differences in usefulness of output.
I think that one can have this view about the “density of progress space” without necessarily postulating much secret sauce.
Jessicata made a comment about phase changes in an earlier part of these discussion writeups. I like the thought of general intelligence as a phase transition. Maybe it’s a phase transition precisely because the progress space is so densely packed at the stage where capabilities generalize. The world has structure. Sufficiently complex or interesting training processes can lay bare this structure. In the “general intelligence as phase transition” model, “stuff that ML does” is the raw horsepower applied to algorithms that exploit the world’s structure. You get immediate benefits from ML-typical “surface-level reasoning.” Then, perhaps there’s a phase transition when the model starts to improve its ways of reasoning and its model buildling as it goes along. This resembles “foom,” but there’s no need for software programming (let alone hardware improvements). The recursion is in the model’s thoughts: Small differences in thinking abilities let the model come up with better learning strategies and plans, which in turn generates compounding benefits over training run. For comparison, small differences between different human babies compound over the human lifetime.
We are in “culture overhang:” The world contains more useful information on how to think better than ever before. But most of the information is useless. If a model is above the threshold where it can filter out useless information and integrate useful information successfully to improve its reasoning, it reaches an attractor with increasing returns.
I tried to argue for this sort of view here. Some quotes from the liked comment:
If the child in the chair next to me in fifth grade was slightly more intellectually curious, somewhat more productive, and marginally better dispositioned to adopt a truth-seeking approach and self-image than I am, this could initially mean they score 100%, and I score 98% on fifth-grade tests – no big difference. But as time goes on, their productivity gets them to read more books, their intellectual curiosity and good judgment get them to read more unusually useful books, and their cleverness gets them to integrate all this knowledge in better and increasingly more creative ways. I’ll reach a point where I’m just sort of skimming things because I’m not motivated enough to understand complicated ideas deeply, whereas they find it rewarding to comprehend everything that gives them a better sense of where to go next on their intellectual journey. By the time we graduate university, my intellectual skills are mostly useless, while they have technical expertise in several topics, can match or even exceed my thinking even on areas I specialized in, and get hired by some leading AI company. The point being: an initially small difference in dispositions becomes almost incomprehensibly vast over time.
[...]
The standard AI foom narrative sounds a bit unrealistic when discussed in terms of some AI system inspecting itself and remodeling its inner architecture in a very deliberate way driven by architectural self-understanding. But what about the framing of being good at learning how to learn? There’s at least a plausible-sounding story we can tell where such an ability might qualify as the “secret sauce” that gives rise to a discontinuity in the returns of increased AI capabilities. In humans – and admittedly this might be too anthropomorphic – I’d think about it in this way: If my 12-year-old self had been brain-uploaded to a suitable virtual reality, made copies of, and given the task of devouring the entire internet in 1,000 years of subjective time (with no aging) to acquire enough knowledge and skill to produce novel and for-the-world useful intellectual contributions, the result probably wouldn’t be much of a success. If we imagined the same with my 19-year-old self, there’s a high chance the result wouldn’t be useful either – but also some chance it would be extremely useful. Assuming, for the sake of the comparison, that a copy clan of 19-year olds can produce highly beneficial research outputs this way, and a copy clan of 12-year olds can’t, what does the landscape look like in between? I don’t find it evident that the in-between is gradual. I think it’s at least plausible that there’s a jump once the copies reach a level of intellectual maturity to make plans which are flexible enough at the meta-level and divide labor sensibly enough to stay open to reassessing their approach as time goes on and they learn new things. Maybe all of that is gradual, and there are degrees of dividing labor sensibly or of staying open to reassessing one’s approach – but that doesn’t seem evident to me. Maybe this works more as an on/off thing.
On this view, it seems natural to assume that small differences in hyperparameter tuning, etc., could have gigantic effects on the resulting AI capabilities once you cross the threshold to general intelligence. Even (perhaps) for the first system that crosses the threshold. We’re in culture overhang and thinking itself is what goes foom once models reach the point where “better thoughts” start changing the nature of thinking.
I’m wondering if it’s a necessary component to an Eliezer-like view on general intelligence and hard takeoff that present-day machine learning methods won’t get us to general intelligence.
I know Eliezer expects there to be algorithmic secret sauce. But another feature of his view seems to be that the “progress space (design-differences-to-usefulness-of-output)” is densely packed at the stage where you reach general intelligence. That’s what his human examples (von Neumann, Jobs) are always driving at: small differences in brain designs generate vast differences in usefulness of output.
I think that one can have this view about the “density of progress space” without necessarily postulating much secret sauce.
Jessicata made a comment about phase changes in an earlier part of these discussion writeups. I like the thought of general intelligence as a phase transition. Maybe it’s a phase transition precisely because the progress space is so densely packed at the stage where capabilities generalize. The world has structure. Sufficiently complex or interesting training processes can lay bare this structure. In the “general intelligence as phase transition” model, “stuff that ML does” is the raw horsepower applied to algorithms that exploit the world’s structure. You get immediate benefits from ML-typical “surface-level reasoning.” Then, perhaps there’s a phase transition when the model starts to improve its ways of reasoning and its model buildling as it goes along. This resembles “foom,” but there’s no need for software programming (let alone hardware improvements). The recursion is in the model’s thoughts: Small differences in thinking abilities let the model come up with better learning strategies and plans, which in turn generates compounding benefits over training run. For comparison, small differences between different human babies compound over the human lifetime.
We are in “culture overhang:” The world contains more useful information on how to think better than ever before. But most of the information is useless. If a model is above the threshold where it can filter out useless information and integrate useful information successfully to improve its reasoning, it reaches an attractor with increasing returns.
I tried to argue for this sort of view here. Some quotes from the liked comment:
On this view, it seems natural to assume that small differences in hyperparameter tuning, etc., could have gigantic effects on the resulting AI capabilities once you cross the threshold to general intelligence. Even (perhaps) for the first system that crosses the threshold. We’re in culture overhang and thinking itself is what goes foom once models reach the point where “better thoughts” start changing the nature of thinking.