Coordination not to build wouldn’t help (even if successful), you can’t defeat an abstract entity, prevent it from doing something in its own abstract world, by preventing existence of its instances in the physical world (intentionally or not), and it can still examine everyone’s motivations and act accordingly. I just suspect that the step of actually building it is a major component of anxiety this seems to produce in some people.
Without the step where an extortion ASI actually gets built, this seems closely analogous to Pascal’s wager (not mugging). There are too many possible abstract entities that act in all sorts of ways in response to all sorts of conditions to make it possible to just point at one of them and have it notice this in an important way. Importance of what happens with all possible abstract entities has to be divided among them, and each of them only gets a little, cashing out as influence of what happens with the entity on what you should do.
So I don’t think there is any reason to expect that any particular arbitrarily selected abstract bogeyman is normatively important for your decision making, because there are all the other abstract bogeymen you are failing to consider. And when you do consider all possible abstract bogeymen, it should just add up to normality.
Then that is a far more salient issue than any acausal blackmail it might have going in its abstract form, which is the only thing that happens in the outcomes where it doesn’t get built (and where it remains unimportant). This just illustrates how the acausal aspects of any of this don’t seem cruxy/relevant, and why I wrote the (top level) answer above the way I did, getting rid of anything acausal from the structure of the problem (other than what acausal structure remains in ordinary coordination among mere humans, guided by shared/overlapping abstract reasons and explanations).
If you can’t affect creation of an extortion ASI, then you can’t affect its posited acausal incentives either, since these things are one and the same.
Within the hypothetical of expecting likely creation of an extortion ASI, what it does and why is no longer unimportant, Pascal’s wager issues no longer apply. Though it still makes sense to remain defiant (to the extent you do have the ability to affect the outcomes), feeding the principle that blackmail works more rarely and that there’s coordination around defying it, maintaining integrity of the worlds that (as a result) remain less affected by its influence.
Comment withdrawn.
Coordination not to build wouldn’t help (even if successful), you can’t defeat an abstract entity, prevent it from doing something in its own abstract world, by preventing existence of its instances in the physical world (intentionally or not), and it can still examine everyone’s motivations and act accordingly. I just suspect that the step of actually building it is a major component of anxiety this seems to produce in some people.
Without the step where an extortion ASI actually gets built, this seems closely analogous to Pascal’s wager (not mugging). There are too many possible abstract entities that act in all sorts of ways in response to all sorts of conditions to make it possible to just point at one of them and have it notice this in an important way. Importance of what happens with all possible abstract entities has to be divided among them, and each of them only gets a little, cashing out as influence of what happens with the entity on what you should do.
So I don’t think there is any reason to expect that any particular arbitrarily selected abstract bogeyman is normatively important for your decision making, because there are all the other abstract bogeymen you are failing to consider. And when you do consider all possible abstract bogeymen, it should just add up to normality.
Comment withdrawn.
Then that is a far more salient issue than any acausal blackmail it might have going in its abstract form, which is the only thing that happens in the outcomes where it doesn’t get built (and where it remains unimportant). This just illustrates how the acausal aspects of any of this don’t seem cruxy/relevant, and why I wrote the (top level) answer above the way I did, getting rid of anything acausal from the structure of the problem (other than what acausal structure remains in ordinary coordination among mere humans, guided by shared/overlapping abstract reasons and explanations).
Comment withdrawn.
If you can’t affect creation of an extortion ASI, then you can’t affect its posited acausal incentives either, since these things are one and the same.
Within the hypothetical of expecting likely creation of an extortion ASI, what it does and why is no longer unimportant, Pascal’s wager issues no longer apply. Though it still makes sense to remain defiant (to the extent you do have the ability to affect the outcomes), feeding the principle that blackmail works more rarely and that there’s coordination around defying it, maintaining integrity of the worlds that (as a result) remain less affected by its influence.