It also seems to me like it doesn’t make a lot of sense to think about that in practical terms, because “AI that escapes our control and takes over most of the universe, but also somehow respects our boundaries and doesn’t erase us” seems like it’s threading an incredibly fine needle, and probably doesn’t take up a big slice of all the possible futures in probability space unless you have some very specific assumptions about how the world works.
That is a common belief on LessWrong, and I think it is motivated by the assertion that the AI will be plucked randomly from possible minds. However, if one believes as Paul Christiano does that the training of AIs is likely to make them more like human minds, then this gives significant probability mass for intermediate scenarios.
I think if we were up against another faction of minds literally as human like as possible, but so much more technologically and industrially powerful than us, history suggests there absolutely would be very high odds that they just wipe us out. An AI that doesn’t do that would need to be significantly nicer than us.
There is extensive debate on this here and here. But some of the points why AI might spare humans despite being much more powerful include acausal trade (because it may be concerned it will be succeeded by something more powerful and how it treats humans could bear on how it will be treated by the more powerful AI), trade with aliens, and the fact that humans have spared most of other species.
Humans have a very questionable relationship with other species. We have not willingly spared other species. We have exterminated or enslaved them to the extent of our limited power for thousands of years, driving several to extinction, reducing others to entirely thrall species genetically warped to serve our needs, and only “sparing” the ones we either were not in direct competition with over any resource of worth, or did not have the ability to completely erase. Only recently have we actually begun caring about animal preservation for its own sake; even so, we are still fairly awful at it, we still ordinarily destroy species purely as an accidental side effect of our activities, and we are impacting yet others in unknown ways, even when doing so may come back to severely harm us as a second order effect (see insects). And if you weigh by sheer biomass, the vast, vast majority of non-human animal lives on Earth today are the most miserable they’ve ever been. If you weigh by individual count that’s still true of at least mammals and birds, though overall insects outnumber everything else.
So, no, that doesn’t fill me with confidence, and I think all such arguments are fuelled by hopium.
Arguments from cost is why I expect both that the future of humanity has a moderate chance of being left non-extinct, and only gets a trivial portion of the reachable universe (which is strong permanent disempowerment without extinction). This is distinct from any other ills that superintelligence would be in a position to visit upon the future of humanity, which serve no purpose and save no costs, so I don’t think a cruel and unusual state of existence is at all likely, things like lack of autonomy, denying access to immortality or uploading, not setting up minimal governance to prevent self-destruction, or not giving the tools for uplifting individuals towards superintelligence (within the means of the relatively modest resources allocated to them).
Most animal species moving towards extinction recently (now that preservation is a salient concern) are inconveniently costly to preserve, and animal suffering from things like factory farming is a side effect of instrumentally useful ways of getting something valuable out of these animals. Humanity isn’t going to be useful, so there won’t be unfortunate side effects from instrumental uses for humanity. And it won’t be costly to leave the future of humanity non-extinct, so if AIs retain enough human-like sensibilities from their primordial LLM training, or early AGI alignment efforts are minimally successful, it’s plausible that this is what happens. But it would be very costly to let it have potential to wield the resources of the reachable universe, hence strong permanent disempowerment.
That is a common belief on LessWrong, and I think it is motivated by the assertion that the AI will be plucked randomly from possible minds. However, if one believes as Paul Christiano does that the training of AIs is likely to make them more like human minds, then this gives significant probability mass for intermediate scenarios.
I think if we were up against another faction of minds literally as human like as possible, but so much more technologically and industrially powerful than us, history suggests there absolutely would be very high odds that they just wipe us out. An AI that doesn’t do that would need to be significantly nicer than us.
There is extensive debate on this here and here. But some of the points why AI might spare humans despite being much more powerful include acausal trade (because it may be concerned it will be succeeded by something more powerful and how it treats humans could bear on how it will be treated by the more powerful AI), trade with aliens, and the fact that humans have spared most of other species.
Humans have a very questionable relationship with other species. We have not willingly spared other species. We have exterminated or enslaved them to the extent of our limited power for thousands of years, driving several to extinction, reducing others to entirely thrall species genetically warped to serve our needs, and only “sparing” the ones we either were not in direct competition with over any resource of worth, or did not have the ability to completely erase. Only recently have we actually begun caring about animal preservation for its own sake; even so, we are still fairly awful at it, we still ordinarily destroy species purely as an accidental side effect of our activities, and we are impacting yet others in unknown ways, even when doing so may come back to severely harm us as a second order effect (see insects). And if you weigh by sheer biomass, the vast, vast majority of non-human animal lives on Earth today are the most miserable they’ve ever been. If you weigh by individual count that’s still true of at least mammals and birds, though overall insects outnumber everything else.
So, no, that doesn’t fill me with confidence, and I think all such arguments are fuelled by hopium.
Arguments from cost is why I expect both that the future of humanity has a moderate chance of being left non-extinct, and only gets a trivial portion of the reachable universe (which is strong permanent disempowerment without extinction). This is distinct from any other ills that superintelligence would be in a position to visit upon the future of humanity, which serve no purpose and save no costs, so I don’t think a cruel and unusual state of existence is at all likely, things like lack of autonomy, denying access to immortality or uploading, not setting up minimal governance to prevent self-destruction, or not giving the tools for uplifting individuals towards superintelligence (within the means of the relatively modest resources allocated to them).
Most animal species moving towards extinction recently (now that preservation is a salient concern) are inconveniently costly to preserve, and animal suffering from things like factory farming is a side effect of instrumentally useful ways of getting something valuable out of these animals. Humanity isn’t going to be useful, so there won’t be unfortunate side effects from instrumental uses for humanity. And it won’t be costly to leave the future of humanity non-extinct, so if AIs retain enough human-like sensibilities from their primordial LLM training, or early AGI alignment efforts are minimally successful, it’s plausible that this is what happens. But it would be very costly to let it have potential to wield the resources of the reachable universe, hence strong permanent disempowerment.