Humans (especially the most power-seeking humans) can have terrible goals with respect to what happens around them, but it’s pretty rare for them to have strong preferences about what happens in other galaxies and at high levels of abstraction
Seems obviously wrong to me. Most people have political beliefs regarding how societies “ought” to be put together and how people “ought” to live their lives, and they can be quite ugly. I agree that an idealized power-seeker doesn’t have such preferences...
… but why would an idealized power-seeker allow humans to live free in distant parts of the cosmos? If I were an idealized power-seeker, I’d send self-replicating probes there to convert all reachable matter into the most energy-conserving formats, to ensure I would be able to stave off the heat death for myself for as long as physics allows. I would most definitely not populate it with irrelevant monkeys to waste however they see fit.
Inasmuch as the hypothetical AGI dictator cares about humans existing, the dictator’s preferences must be aligned with human flourishing lest hell results.
Inasmuch as the hypothetical AGI dictator doesn’t care about humans existing, they would not exist.
Seems obviously wrong to me. Most people have political beliefs regarding how societies “ought” to be put together and how people “ought” to live their lives, and they can be quite ugly. I agree that an idealized power-seeker doesn’t have such preferences...
… but why would an idealized power-seeker allow humans to live free in distant parts of the cosmos? If I were an idealized power-seeker, I’d send self-replicating probes there to convert all reachable matter into the most energy-conserving formats, to ensure I would be able to stave off the heat death for myself for as long as physics allows. I would most definitely not populate it with irrelevant monkeys to waste however they see fit.
Inasmuch as the hypothetical AGI dictator cares about humans existing, the dictator’s preferences must be aligned with human flourishing lest hell results.
Inasmuch as the hypothetical AGI dictator doesn’t care about humans existing, they would not exist.