Maybe they are not explicitly saying “near-optimal”, but it seems to me that they are using models like Solomonoff Induction and AIXI as intuition pumps, and they are getting these beliefs of extreme intelligence from there.
I don’t think anyone at MIRI arrived at worries like ‘AI might be able to deceive their programmers’ or ‘AI might be able to design powerful pathogens’ by staring at the equation for AIXI or AIXItl. AIXI is a useful idea because it’s well-specified enough to let us have conversations that are more than just ‘here are my vague intuitions vs. your vague-intuitions’; it’s math that isn’t quite the right math to directly answer our questions, but at least gets us outside of our own heads, in much the same way that an empirical study can be useful even if it can’t directly answer our questions.
Investigating mathematical and scientific problems that are near to the philosophical problems we care about is a good idea, when we still don’t understand the philosophical problem well enough to directly formalize or test it, because it serves as a point of contact with a domain that isn’t just ‘more vague human intuitions’. Historically this has often been a good way to make intellectual progress, though it’s important to keep in mind just how limited our results are.
AIXI is also useful because the problems we couldn’t solve even if we (impossibly) had recourse to AIXI often overlap with the problems where our theoretical understanding of intelligence is especially lacking, and where we may therefore want to concentrate our early research efforts.
The idea that AI will have various ‘superpowers’ comes more from:
(a) the thought that humans often vary a lot in how much they exhibit the power (without appearing to vary all that much in hardware);
(b) the thought that human brains have known hardware limitations, where existing machines (and a fortiori machines 50 or 100 years down the line) can surpass humans by many orders of magnitude; and
(c) the thought that humans have many unnecessary software limitations, including cases where machines currently outperform humans. There’s also no special reason to expect evolution’s first stab at technology-capable intelligence to have stumbled on all the best possible software ideas.
A more common intuition pump is to simply note that limitations in human brains suggest speed superintelligence is possible, and it’s relatively easy to imagine speed superintelligence allowing one to perform extraordinary feats without imagining other, less well-understood forms of cognitive achievement. Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.
Anyway, do you disagree that MIRI in general expects the kind of low-data, low-experimentation, prior-driven learning that I talked about to be practically possible?
This should be possible to some extent, especially when it comes to progress in mathematics. We should also distinguish software experiments from physical experiments, since it’s a lot harder to keep an AI from performing the former, and the former are much easier to speed up in proportion to speed-ups in the experimenter’s ability to analyze results.
I don’t think there’s any specific consensus view about how much progress requires waiting for results from slow experiments. I frequently hear Luke raise the possibility that slow natural processes could limit rates of self-improvement in AI, but I don’t know whether he considers that a major consideration or a minor one.
I don’t think anyone at MIRI arrived at worries like ‘AI might be able to deceive their programmers’ or ‘AI might be able to design powerful pathogens’ by staring at the equation for AIXI or AIXItl.
In his quantum physics sequence, where he constantly talks (rants, actually) about Solomonoff Induction, Yudkowsky writes:
A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
Mind you, I’m not saying that AIXI is not an interesting and possibly theoretically useful model, my objection is that MIRI people seem to have used it to set a reference class for their intuitions about super-intelligence.
Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.
Extrapolation is always an epistemically questionable endeavor. Intelligence is intrinsically limited by how predictable the world is. Efficiency (time complexity/space complexity/energy complexity/etc.) of algorithms for any computational task is bounded. Hardware resources also have physical limits.
This doesn’t mean that given our current understanding we can claim that human-level intelligence is an upper bound. That would be most likely false. But there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.
This should be possible to some extent, especially when it comes to progress in mathematics.
Ok, but my point is that it has not been established that progress in mathematics will automatically grant an AI “superpowers” in the physical world. And I’d even say that even superpowers by raw cognitive power alone are questionable. Theorem proving can be sped up, but there is more to math than theorem proving.
I think Eliezer mostly just used “Bayesian superintelligence” as a synonym for “superintelligence.” The “Bayesian” is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.
I don’t know as much about Anna’s views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with “AIXI is a theoretical toy. How plausible are smarter systems in the real world?” and proceeds to cite some of the examples I mentioned above, I’m going to guess she isn’t getting most of her intuitions about superintelligence from AIXI either.
there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.
I think our disagreement is about what counts as “extreme” or “extraordinary”, in the “extraordinary claims require extraordinary evidence” sense.
If I’m understanding your perspective, you think we should assume at the outset that humans are about halfway between ‘minimal intelligence’ and ‘maximal intelligence’—a very uninformed prior—and we should then update only very weakly in the direction of ‘humans are closer to the minimum than to the maximum’. Claiming that there’s plenty of room above us is an ‘extreme’ claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are ‘average’, that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.
My view is that we should already have updated strongly away from the ‘humans are average’ prior as soon as we acquired information about how humans arose—through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees’ over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1⁄10 (or for that matter 1⁄10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.
Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than ‘this bird / bomb / brain exists and the rest of the universe is unknown to me’). So although we should not be confident of any specific prediction, ‘humans are near the practical intelligence maximum’ is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not ‘extreme’ in an evidential sense.
We should also distinguish software experiments from physical experiments, since it’s a lot harder to keep an AI from performing the former, and the former are much easier to speed up in proportion to speed-ups in the experimenter’s ability to analyze results.
This is actually completely untrue and is an example of a typical misconception about programming—which is far closer to engineering than math. Every time you compile a program, you are physically testing a theory exactly equivalent to building and testing a physical machine. Every single time you compile and run a program.
If you speedup an AI—by speeding up its mental algorithms or giving it more hardware, you actually slow down the subjective speed of the world and all other software systems in exact proportion. This has enormous consequences—some of which I explored here and here. Human brains operate at 1000hz or or less, which suggests that a near optimal (in terms of raw speed) human-level AGI could run at 1 million X time dilation. However that would effectively mean that the AGI’s computers it had access to would be subjectively slower by 1 million times—so if it’s compiling code for 10 GHZ CPUs, those subjectively run at 10 kilohertz.
I don’t think anyone at MIRI arrived at worries like ‘AI might be able to deceive their programmers’ or ‘AI might be able to design powerful pathogens’ by staring at the equation for AIXI or AIXItl. AIXI is a useful idea because it’s well-specified enough to let us have conversations that are more than just ‘here are my vague intuitions vs. your vague-intuitions’; it’s math that isn’t quite the right math to directly answer our questions, but at least gets us outside of our own heads, in much the same way that an empirical study can be useful even if it can’t directly answer our questions.
Investigating mathematical and scientific problems that are near to the philosophical problems we care about is a good idea, when we still don’t understand the philosophical problem well enough to directly formalize or test it, because it serves as a point of contact with a domain that isn’t just ‘more vague human intuitions’. Historically this has often been a good way to make intellectual progress, though it’s important to keep in mind just how limited our results are.
AIXI is also useful because the problems we couldn’t solve even if we (impossibly) had recourse to AIXI often overlap with the problems where our theoretical understanding of intelligence is especially lacking, and where we may therefore want to concentrate our early research efforts.
The idea that AI will have various ‘superpowers’ comes more from:
(a) the thought that humans often vary a lot in how much they exhibit the power (without appearing to vary all that much in hardware);
(b) the thought that human brains have known hardware limitations, where existing machines (and a fortiori machines 50 or 100 years down the line) can surpass humans by many orders of magnitude; and
(c) the thought that humans have many unnecessary software limitations, including cases where machines currently outperform humans. There’s also no special reason to expect evolution’s first stab at technology-capable intelligence to have stumbled on all the best possible software ideas.
A more common intuition pump is to simply note that limitations in human brains suggest speed superintelligence is possible, and it’s relatively easy to imagine speed superintelligence allowing one to perform extraordinary feats without imagining other, less well-understood forms of cognitive achievement. Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.
This should be possible to some extent, especially when it comes to progress in mathematics. We should also distinguish software experiments from physical experiments, since it’s a lot harder to keep an AI from performing the former, and the former are much easier to speed up in proportion to speed-ups in the experimenter’s ability to analyze results.
I don’t think there’s any specific consensus view about how much progress requires waiting for results from slow experiments. I frequently hear Luke raise the possibility that slow natural processes could limit rates of self-improvement in AI, but I don’t know whether he considers that a major consideration or a minor one.
In his quantum physics sequence, where he constantly talks (rants, actually) about Solomonoff Induction, Yudkowsky writes:
Anna Salamon also mentions AIXI when discussing the feasibility of super-intelligence.
Mind you, I’m not saying that AIXI is not an interesting and possibly theoretically useful model, my objection is that MIRI people seem to have used it to set a reference class for their intuitions about super-intelligence.
Extrapolation is always an epistemically questionable endeavor.
Intelligence is intrinsically limited by how predictable the world is. Efficiency (time complexity/space complexity/energy complexity/etc.) of algorithms for any computational task is bounded. Hardware resources also have physical limits.
This doesn’t mean that given our current understanding we can claim that human-level intelligence is an upper bound. That would be most likely false. But there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.
Ok, but my point is that it has not been established that progress in mathematics will automatically grant an AI “superpowers” in the physical world.
And I’d even say that even superpowers by raw cognitive power alone are questionable. Theorem proving can be sped up, but there is more to math than theorem proving.
I think Eliezer mostly just used “Bayesian superintelligence” as a synonym for “superintelligence.” The “Bayesian” is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.
I don’t know as much about Anna’s views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with “AIXI is a theoretical toy. How plausible are smarter systems in the real world?” and proceeds to cite some of the examples I mentioned above, I’m going to guess she isn’t getting most of her intuitions about superintelligence from AIXI either.
I think our disagreement is about what counts as “extreme” or “extraordinary”, in the “extraordinary claims require extraordinary evidence” sense.
If I’m understanding your perspective, you think we should assume at the outset that humans are about halfway between ‘minimal intelligence’ and ‘maximal intelligence’—a very uninformed prior—and we should then update only very weakly in the direction of ‘humans are closer to the minimum than to the maximum’. Claiming that there’s plenty of room above us is an ‘extreme’ claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are ‘average’, that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.
My view is that we should already have updated strongly away from the ‘humans are average’ prior as soon as we acquired information about how humans arose—through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees’ over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1⁄10 (or for that matter 1⁄10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.
Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than ‘this bird / bomb / brain exists and the rest of the universe is unknown to me’). So although we should not be confident of any specific prediction, ‘humans are near the practical intelligence maximum’ is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not ‘extreme’ in an evidential sense.
This is actually completely untrue and is an example of a typical misconception about programming—which is far closer to engineering than math. Every time you compile a program, you are physically testing a theory exactly equivalent to building and testing a physical machine. Every single time you compile and run a program.
If you speedup an AI—by speeding up its mental algorithms or giving it more hardware, you actually slow down the subjective speed of the world and all other software systems in exact proportion. This has enormous consequences—some of which I explored here and here. Human brains operate at 1000hz or or less, which suggests that a near optimal (in terms of raw speed) human-level AGI could run at 1 million X time dilation. However that would effectively mean that the AGI’s computers it had access to would be subjectively slower by 1 million times—so if it’s compiling code for 10 GHZ CPUs, those subjectively run at 10 kilohertz.