But power seems to be very unrelated to intelligence.
On the level of individuals, perhaps. But one argument is that the more relevant question is that of species-level comparisons; if you need to understand people, to know them, befriend them, network with them, get them to like you, etc., then a human brain may be able to do it, but while a mice or dog brain might manage to do some of it, it’s not going to get to a position of real power that way.
Eliezer making an argument on why one should explicitly not think of “intelligence” as corresponding to conceptual intelligence, but rather to “the thing that makes humans different from other animals”:
General intelligence is a between-species difference, a complex adaptation, and a human universal found in all known cultures. There may as yet be no academic consensus on intelligence, but there is no doubt about the existence, or the power, of the thing-to-be-explained. There is something about humans that let us set our footprints on the Moon.
But the word “intelligence” commonly evokes pictures of the starving professor with an IQ of 160 and the billionaire CEO with an IQ of merely 120. Indeed there are differences of individual ability apart from “book smarts” which contribute to relative success in the human world: enthusiasm, social skills, education, musical talent, rationality. Note that each factor I listed is cognitive. Social skills reside in the brain, not the liver. And jokes aside, you will not find many CEOs, nor yet professors of academia, who are chimpanzees. You will not find many acclaimed rationalists, nor artists, nor poets, nor leaders, nor engineers, nor skilled networkers, nor martial artists, nor musical composers who are mice. Intelligence is the foundation of human power, the strength that fuels our other arts.
The danger of confusing general intelligence with g-factor is that it leads to tremendously underestimating the potential impact of Artificial Intelligence. (This applies to underestimating potential good impacts, as well as potential bad impacts.) Even the phrase “transhuman AI” or “artificial superintelligence” may still evoke images of booksmarts-in-a-box: an AI that’s really good at cognitive tasks stereotypically associated with “intelligence,” like chess or abstract mathematics. But not superhumanly persuasive; or far better than humans at predicting and manipulating human social situations; or inhumanly clever in formulating long-term strategies. So instead of Einstein, should we think of, say, the 19th-century political and diplomatic genius Otto von Bismarck? But that’s only the mirror version of the error. The entire range from village idiot to Einstein, or from village idiot to Bismarck, fits into a small dot on the range from amoeba to human.
If the word “intelligence” evokes Einstein instead of humans, then it may sound sensible to say that intelligence is no match for a gun, as if guns had grown on trees. It may sound sensible to say that intelligence is no match for money, as if mice used money. Human beings didn’t start out with major assets in claws, teeth, armor, or any of the other advantages that were the daily currency of other species. If you had looked at humans from the perspective of the rest of the ecosphere, there was no hint that the squishy things would eventually clothe themselves in armored tanks. We invented the battleground on which we defeated lions and wolves. We did not match them claw for claw, tooth for tooth; we had our own ideas about what mattered. Such is the power of creativity.
On a species level though, the specific niche of human intelligence arose and filled an evolutionary niche, but that is not proof the same strategy will be better.
Bears fill an evolutionary niche of being able to last long times without food, having a wide diet and being very powerful, but that’s not a conclusion that a bear that’s 3x bigger, can eat even more things and can survive even longer without food would fare any better.
Indeed, quite the opposite, if a “better” version of a trait doesn’t exist that likely means the trait is optimized to an extreme.
And in terms of inter-species “achievements”, if the core things that every species want to do is “survive” then, well, it’s fairly easy to conclude cockroaches will outlive us, various grasses will outlive us or at least die with us, same goes for cats… and let’s not even go into exteremophiles, those things might have conquered planets far way from ours billions of years before we even existed, and will certainly outlive us.
Now, our goals obviously converge from those animals, so we think “Oh, poor dumb cockroaches, they shan’t ever advance as a species lacking x/y/z”, but in the umvlet of the cockroach its species has been prospering at an astonishing rate in the most direction that are relevant to it.
Similarly, we are already subpar in many tasks to various algorithms, but that is rather irrelevant since those algorithms aren’t made to fit the niches we do, the very need for them comes from us being unable to fill those niches.
On the level of individuals, perhaps. But one argument is that the more relevant question is that of species-level comparisons; if you need to understand people, to know them, befriend them, network with them, get them to like you, etc., then a human brain may be able to do it, but while a mice or dog brain might manage to do some of it, it’s not going to get to a position of real power that way.
Eliezer making an argument on why one should explicitly not think of “intelligence” as corresponding to conceptual intelligence, but rather to “the thing that makes humans different from other animals”:
On a species level though, the specific niche of human intelligence arose and filled an evolutionary niche, but that is not proof the same strategy will be better.
Bears fill an evolutionary niche of being able to last long times without food, having a wide diet and being very powerful, but that’s not a conclusion that a bear that’s 3x bigger, can eat even more things and can survive even longer without food would fare any better.
Indeed, quite the opposite, if a “better” version of a trait doesn’t exist that likely means the trait is optimized to an extreme.
And in terms of inter-species “achievements”, if the core things that every species want to do is “survive” then, well, it’s fairly easy to conclude cockroaches will outlive us, various grasses will outlive us or at least die with us, same goes for cats… and let’s not even go into exteremophiles, those things might have conquered planets far way from ours billions of years before we even existed, and will certainly outlive us.
Now, our goals obviously converge from those animals, so we think “Oh, poor dumb cockroaches, they shan’t ever advance as a species lacking x/y/z”, but in the umvlet of the cockroach its species has been prospering at an astonishing rate in the most direction that are relevant to it.
Similarly, we are already subpar in many tasks to various algorithms, but that is rather irrelevant since those algorithms aren’t made to fit the niches we do, the very need for them comes from us being unable to fill those niches.