Wouldn’t this imply a bias towards eliminating other agents? (Since that would make the world more predictable, and thereby leave less up to chance?)
A few things to note. Firstly, when I say that there’s a ‘bias’ towards a certain kind of choice, I just mean that the probability that a superintelligent agent with randomly sampled desires (Sia) would make that choice is greater than 1/N, where N is the number of choices available. So, just to emphasize the scale of the effect: even if you were right about that inference, you should still assign very low probability to Sia taking steps to eliminate other agents.
Secondly, when I say that a choice “leaves less up to chance”, I just mean that the sum total of history is more predictable, given that choice, than the sum total of history is predictable, given other choices. (I mention this just because you didn’t read the post, and I want to make sure we’re not talking past each other.)
Thirdly, I would caution against the inference: without humans, things are more predictable; therefore, undertaking to eliminate other agents leaves less up to chance. Even if things are predictable after humans are eliminated, and even if Sia can cook up a foolproof contingency plan for eliminating all humans, that doesn’t mean that that contingency plan leaves less up to chance. Insofar as the contingency plan is sensitive to the human response at various stages, and insofar as that human response is unpredictable (or less predictable than humans are when you don’t try to kill them all), this bias wouldn’t lend any additional probability to Sia choosing that contingency plan.
Fourthly, this bias interacts with the others. Futures without humanity might be futures which involve fewer choices—other deliberative agents tend to force more decisions. So contingency plans which involve human extinction may involve comparatively fewer choicepoints than contingency plans which keep humans around. Insofar as Sia is biased towards contingency plans with more choicepoints, that’s a reason to think she’s biased against eliminating other agents. I don’t have any sense of how these biases interact, or which one is going to be larger in real-world decisions.
Wouldn’t this strongly imply biases towards both self-preservation and resource acquisition?
In some decisions, it may. But I think here, too, we need to tread with caution. In many decisions, this bias makes it somewhat more likely that Sia will pursue self-destruction. To quote myself:
Sia is biased towards choices which allow for more choices—but this isn’t the same thing as being biased towards choices which guarantee more choices. Consider a resolute Sia who is equally likely to choose any contingency plan, and consider the following sequential decision. At stage 1, Sia can either take a ‘safe’ option which will certainly keep her alive or she can play Russian roulette, which has a 1-in-6 probability of killing her. If she takes the ‘safe’ option, the game ends. If she plays Russian roulette and survives, then she’ll once again be given a choice to either take a ‘safe’ option of definitely staying alive or else play Russian roulette. And so on. Whenever she survives a game of Russian roulette, she’s again given the same choice. All else equal, if her desires are sampled normally, a resolute Sia will be much more likely to play Russian roulette at stage 1 than she will be to take the ‘safe’ option.
See the post to understand what I mean by “resolute”—and note that the qualitative effect doesn’t depend upon whether Sia is a resolute chooser.
There are infinitely many desires like that, in fact (that’s what proposition 2 shows).
More generally, take any self-preservation contingency plan, A, and any other contingency plan, B. If we start out uncertain about what Sia wants, then we should think her desires are just as likely to make A more rational than B as they are to make B more rational than A. (That’s what proposition 3 shows.)
That’s rough and subject to a bunch of caveats, of course. I try to go through all of those caveats carefully in the draft.