By thinking about each other’s source code, FAI and Clippy will be able to cooperate acausally like Alice and Bob, each turning their future lightcone into 10% utopia, 90% paperclips. Therefore, we get utopia either way! :D
So even if we lose we win, but even if we win we lose. The amount of utopiastuff is exactly conserved, and launching unaligned AI causes timelines-where-we-win to have less utopia by exactly as much as our timeline has more utopia.
The amount of utopiastuff we get isn’t just proportional to how much we solve alignment, it’s actually back to exactly equal.
Yes, amount of utopiastuff across all worlds remains constant, or possibly even decreases! But I don’t think amount-of-utopiastuff is the thing I want to maximize. I’d love to live in a universe that’s 10% utopia and 90% paperclips! I much prefer that to a 90% chance of extinction and a 10% chance of full-utopia. It’s like insurance. Expected money goes down, but expected utility goes up.
Decision theory does not imply that we get to have nice things, but (I think) it does imply that we get to hedge our insane all-or-nothing gambles for nice things, and redistribute the nice things across more worlds.
So even if we lose we win, but even if we win we lose. The amount of utopiastuff is exactly conserved, and launching unaligned AI causes timelines-where-we-win to have less utopia by exactly as much as our timeline has more utopia.
The amount of utopiastuff we get isn’t just proportional to how much we solve alignment, it’s actually back to exactly equal.
See also: Decision theory does not imply that we get to have nice things.
Yes, amount of utopiastuff across all worlds remains constant, or possibly even decreases! But I don’t think amount-of-utopiastuff is the thing I want to maximize. I’d love to live in a universe that’s 10% utopia and 90% paperclips! I much prefer that to a 90% chance of extinction and a 10% chance of full-utopia. It’s like insurance. Expected money goes down, but expected utility goes up.
Decision theory does not imply that we get to have nice things, but (I think) it does imply that we get to hedge our insane all-or-nothing gambles for nice things, and redistribute the nice things across more worlds.