I think a lot about Homo erectus, which went extinct a bit over 100,000 years ago. They were quite smart, but they were ultimately competing for resources with smarter hominids. Even the chimps and the gorillas mostly survive in tiny numbers, because they didn’t compete too directly with our ancestors, and because we’d be sad if the last ones went extinct. So we make a half-assed attempt to save them.
In the medium-term, intelligence is really valuable! The species that can (let’s say) win a Nobel Prize every six months of GPU time, and copy itself in seconds, is going to rapidly win any evolutionary competition. The world must have been pretty bewildering for Homo erectus towards the end. When the ancestors of the Neaderthals and modern humans showed up, they brought better tools, and much better communication and coordination.
Or to put it another way, the family dog may be loved and pampered, but nobody asks it whether it wants to be spayed, or if the family should buy a new house. The Homo sapiens all sit around a table and make those decisions without the dog’s input. And maybe the only way to put food on the table is to move to the city, and the only good apartment choices forbid pets.
I don’t think the near-term details matter nearly as much as people think. Takeoff in 5 months, 5 years, or 20 years? Incomprehensible alien superintelligences, or vaguely friendly incomprehensible superintelligences who care about us about as much as we care about chimps? A single AI, or competing AIs? The net result is that humans aren’t making the decisions about our future. “Happy humans” have just become a niche luxury good for things that can run rings around us, with their own goals and problems. And those things, since they can replicate and change, will almost certainly face evolutionary forces of their own.
I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?
I see superintelligence a lot like a diagnosis of severe metastatic cancer. It’s not whether it’s going to kill you, it’s a question of how many good years of life you can buy before the end. I support an AI halt, because every year we stay halted buys humans more years of life. I suspect that “alignment” is as futile as Homo erectus trying to “align” Homo sapiens. It’s not completely hopeless—dogs did manage to sort of “align” humans—but I think the absolute best-case scenario is “humans as house pets, with owners of varying quality, likely selecting us to be more pleasing pets.” Which, thank you, but that doesn’t actually sound like success? And all the other possibilities go rapidly downhill from there. And Eliezer would argue that the “house pets” solution is a mirage and he may be right.
I’d prefer to put off rolling those dice as long as we can.
I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?
This is why you won’t build a benevelont godlike singleton until you have vastly more knowledge than we currently have (i.e, with augmenting human intelligence, etc)[1]
I’m not sure I buy the current orientation Eliezer/Nate have to augmented human intelligence in the context of global shutdown, but, does seem like a thing you want before building anything that’s likely to escalate to full overwhelming-in-the-limit-superintelligence.
I think a lot about Homo erectus, which went extinct a bit over 100,000 years ago. They were quite smart, but they were ultimately competing for resources with smarter hominids. Even the chimps and the gorillas mostly survive in tiny numbers, because they didn’t compete too directly with our ancestors, and because we’d be sad if the last ones went extinct. So we make a half-assed attempt to save them.
In the medium-term, intelligence is really valuable! The species that can (let’s say) win a Nobel Prize every six months of GPU time, and copy itself in seconds, is going to rapidly win any evolutionary competition. The world must have been pretty bewildering for Homo erectus towards the end. When the ancestors of the Neaderthals and modern humans showed up, they brought better tools, and much better communication and coordination.
Or to put it another way, the family dog may be loved and pampered, but nobody asks it whether it wants to be spayed, or if the family should buy a new house. The Homo sapiens all sit around a table and make those decisions without the dog’s input. And maybe the only way to put food on the table is to move to the city, and the only good apartment choices forbid pets.
I don’t think the near-term details matter nearly as much as people think. Takeoff in 5 months, 5 years, or 20 years? Incomprehensible alien superintelligences, or vaguely friendly incomprehensible superintelligences who care about us about as much as we care about chimps? A single AI, or competing AIs? The net result is that humans aren’t making the decisions about our future. “Happy humans” have just become a niche luxury good for things that can run rings around us, with their own goals and problems. And those things, since they can replicate and change, will almost certainly face evolutionary forces of their own.
I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?
I see superintelligence a lot like a diagnosis of severe metastatic cancer. It’s not whether it’s going to kill you, it’s a question of how many good years of life you can buy before the end. I support an AI halt, because every year we stay halted buys humans more years of life. I suspect that “alignment” is as futile as Homo erectus trying to “align” Homo sapiens. It’s not completely hopeless—dogs did manage to sort of “align” humans—but I think the absolute best-case scenario is “humans as house pets, with owners of varying quality, likely selecting us to be more pleasing pets.” Which, thank you, but that doesn’t actually sound like success? And all the other possibilities go rapidly downhill from there. And Eliezer would argue that the “house pets” solution is a mirage and he may be right.
I’d prefer to put off rolling those dice as long as we can.
This is why you won’t build a benevelont godlike singleton until you have vastly more knowledge than we currently have (i.e, with augmenting human intelligence, etc)[1]
I’m not sure I buy the current orientation Eliezer/Nate have to augmented human intelligence in the context of global shutdown, but, does seem like a thing you want before building anything that’s likely to escalate to full overwhelming-in-the-limit-superintelligence.