I think Eliezer mostly just used “Bayesian superintelligence” as a synonym for “superintelligence.” The “Bayesian” is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.
I don’t know as much about Anna’s views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with “AIXI is a theoretical toy. How plausible are smarter systems in the real world?” and proceeds to cite some of the examples I mentioned above, I’m going to guess she isn’t getting most of her intuitions about superintelligence from AIXI either.
there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.
I think our disagreement is about what counts as “extreme” or “extraordinary”, in the “extraordinary claims require extraordinary evidence” sense.
If I’m understanding your perspective, you think we should assume at the outset that humans are about halfway between ‘minimal intelligence’ and ‘maximal intelligence’—a very uninformed prior—and we should then update only very weakly in the direction of ‘humans are closer to the minimum than to the maximum’. Claiming that there’s plenty of room above us is an ‘extreme’ claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are ‘average’, that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.
My view is that we should already have updated strongly away from the ‘humans are average’ prior as soon as we acquired information about how humans arose—through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees’ over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1⁄10 (or for that matter 1⁄10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.
Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than ‘this bird / bomb / brain exists and the rest of the universe is unknown to me’). So although we should not be confident of any specific prediction, ‘humans are near the practical intelligence maximum’ is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not ‘extreme’ in an evidential sense.
I think Eliezer mostly just used “Bayesian superintelligence” as a synonym for “superintelligence.” The “Bayesian” is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.
I don’t know as much about Anna’s views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with “AIXI is a theoretical toy. How plausible are smarter systems in the real world?” and proceeds to cite some of the examples I mentioned above, I’m going to guess she isn’t getting most of her intuitions about superintelligence from AIXI either.
I think our disagreement is about what counts as “extreme” or “extraordinary”, in the “extraordinary claims require extraordinary evidence” sense.
If I’m understanding your perspective, you think we should assume at the outset that humans are about halfway between ‘minimal intelligence’ and ‘maximal intelligence’—a very uninformed prior—and we should then update only very weakly in the direction of ‘humans are closer to the minimum than to the maximum’. Claiming that there’s plenty of room above us is an ‘extreme’ claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are ‘average’, that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.
My view is that we should already have updated strongly away from the ‘humans are average’ prior as soon as we acquired information about how humans arose—through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees’ over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1⁄10 (or for that matter 1⁄10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.
Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than ‘this bird / bomb / brain exists and the rest of the universe is unknown to me’). So although we should not be confident of any specific prediction, ‘humans are near the practical intelligence maximum’ is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not ‘extreme’ in an evidential sense.