Suppose that I had a plan that will wipe out humanity with probability 0.999999999, but will lead to a positive Singularity with probability 0.000000001 minus epsilon. I understand why that would be a terrible plan
That means that the utility of a positive singularity (on a scale where the status quo is at 0) is less than −10^9 times the utility of wiping out humanity.
Now suppose I had a plan that will wipe out 0.999999999 of humanity’s measure, but lead to a positive Singularity in all but epsilon of the remaining measure … On reflection, after understanding better, would we conclude that this is also a terrible plan? I strongly suspect so, but I can not explain exactly why it would be.
This is left as an exercise. Hint: the previous paragraph of mine may be relevant.
That means that the utility of a positive singularity (on a scale where the status quo is at 0) is less than −10^9 times the utility of wiping out humanity.
This is left as an exercise. Hint: the previous paragraph of mine may be relevant.