I think this is correct if your model of quality-of-values is based on comparing virtue, but incorrect when you account for scope, distance, and human-ness. Humans (especially the most power-seeking humans) can have terrible goals with respect to what happens around them, but it’s pretty rare for them to have strong preferences about what happens in other galaxies and at high levels of abstraction. And most poeple have values that require other people to at least exist (with significant less risk of philosophical trickery in which something nonsentient gets mis-counted as humans).
I think the failure case for a human takeover is probably that most of the universe is pretty good, the areas that can communicate with the dictator without long light speed delays are worse, and the areas that the dictator observes directly is bad. In order for the whole universe to be bad, the dictator would need to have strong preferences about parts of the universe that he’ll never get to see, which requires a philosophical mindset which I think is quite negatively correlated with that sort of power seeking.
Humans (especially the most power-seeking humans) can have terrible goals with respect to what happens around them, but it’s pretty rare for them to have strong preferences about what happens in other galaxies and at high levels of abstraction
Seems obviously wrong to me. Most people have political beliefs regarding how societies “ought” to be put together and how people “ought” to live their lives, and they can be quite ugly. I agree that an idealized power-seeker doesn’t have such preferences...
… but why would an idealized power-seeker allow humans to live free in distant parts of the cosmos? If I were an idealized power-seeker, I’d send self-replicating probes there to convert all reachable matter into the most energy-conserving formats, to ensure I would be able to stave off the heat death for myself for as long as physics allows. I would most definitely not populate it with irrelevant monkeys to waste however they see fit.
Inasmuch as the hypothetical AGI dictator cares about humans existing, the dictator’s preferences must be aligned with human flourishing lest hell results.
Inasmuch as the hypothetical AGI dictator doesn’t care about humans existing, they would not exist.
I think this is correct if your model of quality-of-values is based on comparing virtue, but incorrect when you account for scope, distance, and human-ness. Humans (especially the most power-seeking humans) can have terrible goals with respect to what happens around them, but it’s pretty rare for them to have strong preferences about what happens in other galaxies and at high levels of abstraction. And most poeple have values that require other people to at least exist (with significant less risk of philosophical trickery in which something nonsentient gets mis-counted as humans).
I think the failure case for a human takeover is probably that most of the universe is pretty good, the areas that can communicate with the dictator without long light speed delays are worse, and the areas that the dictator observes directly is bad. In order for the whole universe to be bad, the dictator would need to have strong preferences about parts of the universe that he’ll never get to see, which requires a philosophical mindset which I think is quite negatively correlated with that sort of power seeking.
Seems obviously wrong to me. Most people have political beliefs regarding how societies “ought” to be put together and how people “ought” to live their lives, and they can be quite ugly. I agree that an idealized power-seeker doesn’t have such preferences...
… but why would an idealized power-seeker allow humans to live free in distant parts of the cosmos? If I were an idealized power-seeker, I’d send self-replicating probes there to convert all reachable matter into the most energy-conserving formats, to ensure I would be able to stave off the heat death for myself for as long as physics allows. I would most definitely not populate it with irrelevant monkeys to waste however they see fit.
Inasmuch as the hypothetical AGI dictator cares about humans existing, the dictator’s preferences must be aligned with human flourishing lest hell results.
Inasmuch as the hypothetical AGI dictator doesn’t care about humans existing, they would not exist.