This is definitely a split which I think underlies a lot of differing intuitions about AGI and timelines. That said, the versions of each which are compatible with evidence/constraints generally have similar implications for at least the basics of AI risk (though they differ in predictions about what AI looks like “later on”, once it’s already far past eclipsing the capabilities of the human species).
Key relevant evidence/constraints, under my usual framing:
We live in a very high dimensional environment. When doing science/optimization in such an environment, brute-force is search is exponentially intractable, so having e.g. ten billion humans running the same basic brute-force algorithm will not be qualitatively better than one human running a brute-force algorithm. The fact that less-than-exponentially-large numbers of humans are able to perform as well as we are implies that there’s some real “general intelligence” going on in there somewhere.
That said, it’s still possible-in-principle for whatever general intelligence we have to be importantly distributed across humans. What the dimensionality argument rules out is a model in which humans’ capabilities are just about brute-force trying lots of stuff, and then memetic spread of whatever works. The “trying stuff” step has to be doing “most of the work”, in some sense, of finding good models/techniques/etc; but whatever process is doing that work could itself be load-bearingly spread across humans.
Also, memetic spread could still be a bottleneck in practice, even if it’s not “doing most of the work” in an algorithmic sense.
A lower bound for what AI can do is “run lots of human-equivalent minds, and cheaply copy them”. Even under a model where memetic spread is the main bottlenecking step for humans, AI will still be ridiculously better at that. You know that problem humans have where we spend tons of effort accumulating “tacit knowledge” which is hard to convey to the next generation? For AI, cheap copy means that problem is just completely gone.
Humans’ own historical progress/experience puts an upper bound on how hard it is to solve novel problems (not solved by society today). Humans have done… rather ridiculously a lot of that, over the past 250 years. That, in turn, lower bounds what AIs will be capable of.
I’d like to see more discussion of this, I read some of the FOOM debate but I’m assuming that there has been more discussion of this important issue since?
I suppose the key question is for recursive self-improvement. We can give hardware improvement (improved hardware allows design of more complex and better hardware) because we are on the treadmill already. But how likely is algorithmic self-improvement. For an intelligence to be able to improve itself algorithmically the following seem to need to hold.
The system needs to understand itself
There has to be some capacity that can be improved without detriment to some other capacity (else you are doing some self-optimization and not necessarily improvement)
If it is the memeplex that gives us our generality (as is suggested by our flowering of discovery over the past 250 years compared to the past 300,000 years of homo sapiens), it might not be understandable. It would be in the weights or equivalents in whatever the AI uses. No human would understand it either.
Fiddling about with weights without knowledge would likely lead to trade offs and so you might not have the second consideration holding.
I’m not saying AI won’t change history, but we need an accurate view of how it will change things.
On the matter of software improvements potentially available during recursive self-improvement, we can look at the current pace of algorithmic improvement, which has been probably faster than scaling for some time now. So that’s another lower bound on what AI will be capable of, assuming that the extrapolation holds up.
I’m wary about that one, because that isn’t a known “general” intelligence architecture, so we can expect AIs to make better learning algorithms for deep neural networks, but not necessarily themselves.
This is definitely a split which I think underlies a lot of differing intuitions about AGI and timelines. That said, the versions of each which are compatible with evidence/constraints generally have similar implications for at least the basics of AI risk (though they differ in predictions about what AI looks like “later on”, once it’s already far past eclipsing the capabilities of the human species).
Key relevant evidence/constraints, under my usual framing:
We live in a very high dimensional environment. When doing science/optimization in such an environment, brute-force is search is exponentially intractable, so having e.g. ten billion humans running the same basic brute-force algorithm will not be qualitatively better than one human running a brute-force algorithm. The fact that less-than-exponentially-large numbers of humans are able to perform as well as we are implies that there’s some real “general intelligence” going on in there somewhere.
That said, it’s still possible-in-principle for whatever general intelligence we have to be importantly distributed across humans. What the dimensionality argument rules out is a model in which humans’ capabilities are just about brute-force trying lots of stuff, and then memetic spread of whatever works. The “trying stuff” step has to be doing “most of the work”, in some sense, of finding good models/techniques/etc; but whatever process is doing that work could itself be load-bearingly spread across humans.
Also, memetic spread could still be a bottleneck in practice, even if it’s not “doing most of the work” in an algorithmic sense.
A lower bound for what AI can do is “run lots of human-equivalent minds, and cheaply copy them”. Even under a model where memetic spread is the main bottlenecking step for humans, AI will still be ridiculously better at that. You know that problem humans have where we spend tons of effort accumulating “tacit knowledge” which is hard to convey to the next generation? For AI, cheap copy means that problem is just completely gone.
Humans’ own historical progress/experience puts an upper bound on how hard it is to solve novel problems (not solved by society today). Humans have done… rather ridiculously a lot of that, over the past 250 years. That, in turn, lower bounds what AIs will be capable of.
I’d like to see more discussion of this, I read some of the FOOM debate but I’m assuming that there has been more discussion of this important issue since?
I suppose the key question is for recursive self-improvement. We can give hardware improvement (improved hardware allows design of more complex and better hardware) because we are on the treadmill already. But how likely is algorithmic self-improvement. For an intelligence to be able to improve itself algorithmically the following seem to need to hold.
The system needs to understand itself
There has to be some capacity that can be improved without detriment to some other capacity (else you are doing some self-optimization and not necessarily improvement)
If it is the memeplex that gives us our generality (as is suggested by our flowering of discovery over the past 250 years compared to the past 300,000 years of homo sapiens), it might not be understandable. It would be in the weights or equivalents in whatever the AI uses. No human would understand it either.
Fiddling about with weights without knowledge would likely lead to trade offs and so you might not have the second consideration holding.
I’m not saying AI won’t change history, but we need an accurate view of how it will change things.
On the matter of software improvements potentially available during recursive self-improvement, we can look at the current pace of algorithmic improvement, which has been probably faster than scaling for some time now. So that’s another lower bound on what AI will be capable of, assuming that the extrapolation holds up.
I’m wary about that one, because that isn’t a known “general” intelligence architecture, so we can expect AIs to make better learning algorithms for deep neural networks, but not necessarily themselves.