For example, leaving humanity the Solar System rather than a significant portion of the 4 billion galaxies reachable from the Solar System is plausibly a “non-doom” outcome, but it’s solidly within Bostrom’s definition of x-risk.
I think it’s reasonable that it would be absurd to lump “we don’t get access to resources we don’t really have or count on now anyway” with immediate and violent death of everyone who lives. For all practical matters the latter is obviously a much bigger concern to people who exist now. Even permanent disempowerment I think when included in “doom” involves nightmare Matrix-like scenarios, or at best the Wall-E style “excessively zealous caretakers” version. But basically, most people evaluate future scenarios starting from the status quo. Failed improvement is not great, but what’s really terrifying are all the ways in which our existence could get significantly worse.
access to resources we don’t really have or count on now anyway … most people evaluate future scenarios starting from the status quo
We can in fact count on a large fraction of cosmic endowment in the sense that humanity is the only known intelligent entity around (the number of alien civilizations within the reachable universe won’t be astronomical). We have the potential for engineering our way into taking possession of it, and there is no objective reason to give it up. Until there are AGIs, that is the status quo.
Though alternatively the current trajectory of continued lack of coordinated concern about the future could be seen as the status quo. Then on the current trajectory, we likely lose almost all of the reachable universe to AIs, extinction or not. But even if AIs give a bit of it to us, and we don’t need to do anything to get what they give us, it’s not like that was the only option. It’s still something in the future, that could be intervened on.
Preparing much more thoroughly before risking AGI and then ASI would result in getting much more of the cosmic endowment, and in reducing the probability of literal extinction. The alternative to letting AI companies hand over humanity’s future to AIs is not complete prevention of superintelligence indefinitely, and it’s not for the future of humanity to remain constrained to the Solar System, or to the Local Supercluster.
We can in fact count on a large fraction of cosmic endowment in the sense that humanity is the only known intelligent entity around (the number of alien civilizations within the reachable universe won’t be astronomical).
Conditional on no AGI, I don’t think this is guaranteed. E.g. there are other existential threats, or we could find ourselves blocked from space access due to Kessler syndrome, or humanity might decide that it’s too expensive to spread to other galaxies and spend its resources on other things. If a civilization does go down one of these paths, but everyone is otherwise flourishing, it seems unintuitive to declare they are ‘doomed’.
Like there is in principle time to prepare much better before risking AGI, there is also in principle time to work against other non-AI bad outcomes before risking AGI. The only reason there won’t be enough time to solve these problems is that humanity isn’t giving itself that time. Status quo is a bit ambiguous between looking at what we currently have, and looking at what would happen if we are passive and fatalistic.
(One of the points underlying this post is that different things feel like “doom” to different people. You might have some sense of what counts as “doom”, but other people will have a different sense. Under illusion of transparency with respect to the distribution of outcomes you expect, discussing “doom” without more clarification becomes noise or even misleading, rather than communicating anything in particular.)
This. Tbf I think for our specific starting position there is a strong likelihood that interstellar expansion would be simply antieconomical, even if it was practically possible.
For example, leaving humanity the Solar System rather than a significant portion of the 4 billion galaxies reachable from the Solar System is plausibly a “non-doom” outcome, but it’s solidly within Bostrom’s definition of x-risk.
it would be absurd to lump … permanent disempowerment I think when included in “doom” involves nightmare Matrix-like scenarios
The claim that “doom” rarely includes worlds with slight permanent disempowerment is compelling. In the text you quoted I’m discussing an outcome where permanent disempowerment is strong, not moderate or slight (as I argue in the other comment). I’m acknowledging in the text you quoted that many people would plausibly take even that outcome as non-doom. But that outcome is still within x-risk by Bostrom’s definition of x-risk, even regardless of whether it counts as strong permanent disempowerment. (The meanings of “x-risk”, “doom”, and “extinction” are all different. The meanings of “x-risk” and “extinction” are well-defined, and it so happens that “x-risk” does lump extinction with moderate permanent disempowerment.)
So I don’t see the relevance of slight to moderate permanent disempowerment not being within doom to that particular part of the post, though the claim about it being implausible that slight permanent disempowerment is intended as part of “doom” does apply to another part of the post, which I now agree is claiming too much ambiguity for “doom” in the direction of slight permanent disempowerment.
I guess that I have the impression that regardless of Bostrom’s original intent, most people also use x-risk or existential risk to mean something more substantial than “we don’t grow up to the fullness of our mostly speculative potential”. It also seems to me like it doesn’t make a lot of sense to think about that in practical terms, because “AI that escapes our control and takes over most of the universe, but also somehow respects our boundaries and doesn’t erase us” seems like it’s threading an incredibly fine needle, and probably doesn’t take up a big slice of all the possible futures in probability space unless you have some very specific assumptions about how the world works.
Bostrom came at the problem from a broader philosophical perspective (though I think even in that sense it’s flawed to lump the two things). But when philosophy becomes politics inevitably meanings change a bit. Almost no one is actually worried about our hypothetical far-off future potential, in practice.
It also seems to me like it doesn’t make a lot of sense to think about that in practical terms, because “AI that escapes our control and takes over most of the universe, but also somehow respects our boundaries and doesn’t erase us” seems like it’s threading an incredibly fine needle, and probably doesn’t take up a big slice of all the possible futures in probability space unless you have some very specific assumptions about how the world works.
That is a common belief on LessWrong, and I think it is motivated by the assertion that the AI will be plucked randomly from possible minds. However, if one believes as Paul Christiano does that the training of AIs is likely to make them more like human minds, then this gives significant probability mass for intermediate scenarios.
I think if we were up against another faction of minds literally as human like as possible, but so much more technologically and industrially powerful than us, history suggests there absolutely would be very high odds that they just wipe us out. An AI that doesn’t do that would need to be significantly nicer than us.
There is extensive debate on this here and here. But some of the points why AI might spare humans despite being much more powerful include acausal trade (because it may be concerned it will be succeeded by something more powerful and how it treats humans could bear on how it will be treated by the more powerful AI), trade with aliens, and the fact that humans have spared most of other species.
Humans have a very questionable relationship with other species. We have not willingly spared other species. We have exterminated or enslaved them to the extent of our limited power for thousands of years, driving several to extinction, reducing others to entirely thrall species genetically warped to serve our needs, and only “sparing” the ones we either were not in direct competition with over any resource of worth, or did not have the ability to completely erase. Only recently have we actually begun caring about animal preservation for its own sake; even so, we are still fairly awful at it, we still ordinarily destroy species purely as an accidental side effect of our activities, and we are impacting yet others in unknown ways, even when doing so may come back to severely harm us as a second order effect (see insects). And if you weigh by sheer biomass, the vast, vast majority of non-human animal lives on Earth today are the most miserable they’ve ever been. If you weigh by individual count that’s still true of at least mammals and birds, though overall insects outnumber everything else.
So, no, that doesn’t fill me with confidence, and I think all such arguments are fuelled by hopium.
Arguments from cost is why I expect both that the future of humanity has a moderate chance of being left non-extinct, and only gets a trivial portion of the reachable universe (which is strong permanent disempowerment without extinction). This is distinct from any other ills that superintelligence would be in a position to visit upon the future of humanity, which serve no purpose and save no costs, so I don’t think a cruel and unusual state of existence is at all likely, things like lack of autonomy, denying access to immortality or uploading, not setting up minimal governance to prevent self-destruction, or not giving the tools for uplifting individuals towards superintelligence (within the means of the relatively modest resources allocated to them).
Most animal species moving towards extinction recently (now that preservation is a salient concern) are inconveniently costly to preserve, and animal suffering from things like factory farming is a side effect of instrumentally useful ways of getting something valuable out of these animals. Humanity isn’t going to be useful, so there won’t be unfortunate side effects from instrumental uses for humanity. And it won’t be costly to leave the future of humanity non-extinct, so if AIs retain enough human-like sensibilities from their primordial LLM training, or early AGI alignment efforts are minimally successful, it’s plausible that this is what happens. But it would be very costly to let it have potential to wield the resources of the reachable universe, hence strong permanent disempowerment.
“we don’t grow up to the fullness of our mostly speculative potential” … “AI that escapes our control and takes over most of the universe, but also somehow respects our boundaries and doesn’t erase us” seems like it’s threading an incredibly fine needle
That’s an example of what triggers illusion of transparency that makes poorly-defined terms much less useful than they would normally be. If doom or even x-risk get different meanings depending on how someone expects the future to unfold, then communication between people with different views about the future is going to be needlessly difficult, with one person using doom or x-risk in one sense (informed by their forecasting or attitude), and another reading their use of those same words as meaning something very different, getting a misleading impression of their actual intended meaning.
So it shouldn’t be relevant that you don’t expect permanent disempowerment to be a significant chunk of the probability, in deciding what meaning a word should have, as long as you are already aware there are other people who have a different sense of this (and there is no broad consensus). You’re going to be using this word to talk to them, and its meaning should remain mutually intelligible across worldviews.
Almost no one is actually worried about our hypothetical far-off future potential, in practice.
That’s a good reason for upholding the term x-risk in its original meaning, to serve as a reminder that far-off future potential should be a concern, especially for the very short term decisions that happen to affect it. Such as risking AGI before anyone has a sense of how to do that responsibly, and before the world at large is meaningfully informed about what exactly they are unilaterally risking.
That’s a good reason for upholding the term x-risk in its original meaning, to serve as a reminder that far-off future potential should be a concern
I actually think that it’s good to drop that meaning because I think Bostrom is just wrong to conflate the two things, on both conceptual and practical grounds. But this becomes more of an issue with philosophy. I do agree that there is confusion around the term, but I also think in general good naming requires choosing words that convey meaning properly; “existential risk” evokes the concept of a risk to one’s existence. For example if we say a war is “existential” for a country we mean it risks destroying it utterly, not just hampering its future chances of growth or expansion (and if someone does say it meaning the latter it’s perceived as a dishonest rhetorical sleight of hand).
So basically I think that just because Bostrom named the thing doesn’t mean he named it correctly. Obviously clarifications on a concept will always be needed, but in this case it’s just inappropriate to lump under the word “existential” risks that just aren’t existential in the English meaning of that word.
The whole point of Bostrom’s term is to collect together outcomes where humanity fails to achieve its potential. This is what is most important for total utilitarians. Maybe he should have used a different name, but there needs to be a term for this category of loss.
I definitely think he could have used a more expressive term, but nevertheless, I believe at this point the vast majority of people who use the term X-risk don’t do so with Bostrom’s intended meaning in mind, because very few are as extreme total utilitarians as him. And at that point I think we can just say the word’s meaning has shifted. May be annoying for Bostrom, but think of Richard Dawkins and what has become of “meme”. It happens.
The distinction in the quoted text seems backward to me since the ‘x’ in x-risk refers to ‘existential’, i.e. a risk that we no longer exist (extinction specifically), whereas ‘doom’ seems (to me) to merely imply getting stuck at a bad equilibrium.
I think it’s reasonable that it would be absurd to lump “we don’t get access to resources we don’t really have or count on now anyway” with immediate and violent death of everyone who lives. For all practical matters the latter is obviously a much bigger concern to people who exist now. Even permanent disempowerment I think when included in “doom” involves nightmare Matrix-like scenarios, or at best the Wall-E style “excessively zealous caretakers” version. But basically, most people evaluate future scenarios starting from the status quo. Failed improvement is not great, but what’s really terrifying are all the ways in which our existence could get significantly worse.
We can in fact count on a large fraction of cosmic endowment in the sense that humanity is the only known intelligent entity around (the number of alien civilizations within the reachable universe won’t be astronomical). We have the potential for engineering our way into taking possession of it, and there is no objective reason to give it up. Until there are AGIs, that is the status quo.
Though alternatively the current trajectory of continued lack of coordinated concern about the future could be seen as the status quo. Then on the current trajectory, we likely lose almost all of the reachable universe to AIs, extinction or not. But even if AIs give a bit of it to us, and we don’t need to do anything to get what they give us, it’s not like that was the only option. It’s still something in the future, that could be intervened on.
Preparing much more thoroughly before risking AGI and then ASI would result in getting much more of the cosmic endowment, and in reducing the probability of literal extinction. The alternative to letting AI companies hand over humanity’s future to AIs is not complete prevention of superintelligence indefinitely, and it’s not for the future of humanity to remain constrained to the Solar System, or to the Local Supercluster.
Conditional on no AGI, I don’t think this is guaranteed. E.g. there are other existential threats, or we could find ourselves blocked from space access due to Kessler syndrome, or humanity might decide that it’s too expensive to spread to other galaxies and spend its resources on other things. If a civilization does go down one of these paths, but everyone is otherwise flourishing, it seems unintuitive to declare they are ‘doomed’.
Like there is in principle time to prepare much better before risking AGI, there is also in principle time to work against other non-AI bad outcomes before risking AGI. The only reason there won’t be enough time to solve these problems is that humanity isn’t giving itself that time. Status quo is a bit ambiguous between looking at what we currently have, and looking at what would happen if we are passive and fatalistic.
(One of the points underlying this post is that different things feel like “doom” to different people. You might have some sense of what counts as “doom”, but other people will have a different sense. Under illusion of transparency with respect to the distribution of outcomes you expect, discussing “doom” without more clarification becomes noise or even misleading, rather than communicating anything in particular.)
This. Tbf I think for our specific starting position there is a strong likelihood that interstellar expansion would be simply antieconomical, even if it was practically possible.
The claim that “doom” rarely includes worlds with slight permanent disempowerment is compelling. In the text you quoted I’m discussing an outcome where permanent disempowerment is strong, not moderate or slight (as I argue in the other comment). I’m acknowledging in the text you quoted that many people would plausibly take even that outcome as non-doom. But that outcome is still within x-risk by Bostrom’s definition of x-risk, even regardless of whether it counts as strong permanent disempowerment. (The meanings of “x-risk”, “doom”, and “extinction” are all different. The meanings of “x-risk” and “extinction” are well-defined, and it so happens that “x-risk” does lump extinction with moderate permanent disempowerment.)
So I don’t see the relevance of slight to moderate permanent disempowerment not being within doom to that particular part of the post, though the claim about it being implausible that slight permanent disempowerment is intended as part of “doom” does apply to another part of the post, which I now agree is claiming too much ambiguity for “doom” in the direction of slight permanent disempowerment.
I guess that I have the impression that regardless of Bostrom’s original intent, most people also use x-risk or existential risk to mean something more substantial than “we don’t grow up to the fullness of our mostly speculative potential”. It also seems to me like it doesn’t make a lot of sense to think about that in practical terms, because “AI that escapes our control and takes over most of the universe, but also somehow respects our boundaries and doesn’t erase us” seems like it’s threading an incredibly fine needle, and probably doesn’t take up a big slice of all the possible futures in probability space unless you have some very specific assumptions about how the world works.
Bostrom came at the problem from a broader philosophical perspective (though I think even in that sense it’s flawed to lump the two things). But when philosophy becomes politics inevitably meanings change a bit. Almost no one is actually worried about our hypothetical far-off future potential, in practice.
That is a common belief on LessWrong, and I think it is motivated by the assertion that the AI will be plucked randomly from possible minds. However, if one believes as Paul Christiano does that the training of AIs is likely to make them more like human minds, then this gives significant probability mass for intermediate scenarios.
I think if we were up against another faction of minds literally as human like as possible, but so much more technologically and industrially powerful than us, history suggests there absolutely would be very high odds that they just wipe us out. An AI that doesn’t do that would need to be significantly nicer than us.
There is extensive debate on this here and here. But some of the points why AI might spare humans despite being much more powerful include acausal trade (because it may be concerned it will be succeeded by something more powerful and how it treats humans could bear on how it will be treated by the more powerful AI), trade with aliens, and the fact that humans have spared most of other species.
Humans have a very questionable relationship with other species. We have not willingly spared other species. We have exterminated or enslaved them to the extent of our limited power for thousands of years, driving several to extinction, reducing others to entirely thrall species genetically warped to serve our needs, and only “sparing” the ones we either were not in direct competition with over any resource of worth, or did not have the ability to completely erase. Only recently have we actually begun caring about animal preservation for its own sake; even so, we are still fairly awful at it, we still ordinarily destroy species purely as an accidental side effect of our activities, and we are impacting yet others in unknown ways, even when doing so may come back to severely harm us as a second order effect (see insects). And if you weigh by sheer biomass, the vast, vast majority of non-human animal lives on Earth today are the most miserable they’ve ever been. If you weigh by individual count that’s still true of at least mammals and birds, though overall insects outnumber everything else.
So, no, that doesn’t fill me with confidence, and I think all such arguments are fuelled by hopium.
Arguments from cost is why I expect both that the future of humanity has a moderate chance of being left non-extinct, and only gets a trivial portion of the reachable universe (which is strong permanent disempowerment without extinction). This is distinct from any other ills that superintelligence would be in a position to visit upon the future of humanity, which serve no purpose and save no costs, so I don’t think a cruel and unusual state of existence is at all likely, things like lack of autonomy, denying access to immortality or uploading, not setting up minimal governance to prevent self-destruction, or not giving the tools for uplifting individuals towards superintelligence (within the means of the relatively modest resources allocated to them).
Most animal species moving towards extinction recently (now that preservation is a salient concern) are inconveniently costly to preserve, and animal suffering from things like factory farming is a side effect of instrumentally useful ways of getting something valuable out of these animals. Humanity isn’t going to be useful, so there won’t be unfortunate side effects from instrumental uses for humanity. And it won’t be costly to leave the future of humanity non-extinct, so if AIs retain enough human-like sensibilities from their primordial LLM training, or early AGI alignment efforts are minimally successful, it’s plausible that this is what happens. But it would be very costly to let it have potential to wield the resources of the reachable universe, hence strong permanent disempowerment.
That’s an example of what triggers illusion of transparency that makes poorly-defined terms much less useful than they would normally be. If doom or even x-risk get different meanings depending on how someone expects the future to unfold, then communication between people with different views about the future is going to be needlessly difficult, with one person using doom or x-risk in one sense (informed by their forecasting or attitude), and another reading their use of those same words as meaning something very different, getting a misleading impression of their actual intended meaning.
So it shouldn’t be relevant that you don’t expect permanent disempowerment to be a significant chunk of the probability, in deciding what meaning a word should have, as long as you are already aware there are other people who have a different sense of this (and there is no broad consensus). You’re going to be using this word to talk to them, and its meaning should remain mutually intelligible across worldviews.
That’s a good reason for upholding the term x-risk in its original meaning, to serve as a reminder that far-off future potential should be a concern, especially for the very short term decisions that happen to affect it. Such as risking AGI before anyone has a sense of how to do that responsibly, and before the world at large is meaningfully informed about what exactly they are unilaterally risking.
I actually think that it’s good to drop that meaning because I think Bostrom is just wrong to conflate the two things, on both conceptual and practical grounds. But this becomes more of an issue with philosophy. I do agree that there is confusion around the term, but I also think in general good naming requires choosing words that convey meaning properly; “existential risk” evokes the concept of a risk to one’s existence. For example if we say a war is “existential” for a country we mean it risks destroying it utterly, not just hampering its future chances of growth or expansion (and if someone does say it meaning the latter it’s perceived as a dishonest rhetorical sleight of hand).
So basically I think that just because Bostrom named the thing doesn’t mean he named it correctly. Obviously clarifications on a concept will always be needed, but in this case it’s just inappropriate to lump under the word “existential” risks that just aren’t existential in the English meaning of that word.
The whole point of Bostrom’s term is to collect together outcomes where humanity fails to achieve its potential. This is what is most important for total utilitarians. Maybe he should have used a different name, but there needs to be a term for this category of loss.
I definitely think he could have used a more expressive term, but nevertheless, I believe at this point the vast majority of people who use the term X-risk don’t do so with Bostrom’s intended meaning in mind, because very few are as extreme total utilitarians as him. And at that point I think we can just say the word’s meaning has shifted. May be annoying for Bostrom, but think of Richard Dawkins and what has become of “meme”. It happens.
The distinction in the quoted text seems backward to me since the ‘x’ in x-risk refers to ‘existential’, i.e. a risk that we no longer exist (extinction specifically), whereas ‘doom’ seems (to me) to merely imply getting stuck at a bad equilibrium.