An issue here re the CEV of powerful people like Vladimir Putin not being that bad compared to nothingness is that currently, even dictators require most people in at least somewhat functional states in order to have power, and more generally the generosity of capitalism is at least in part based on humans being useful when fed and educated.
But in a world where AI automates away human labor, the incentives for powerful people (absent even mild values of compassion/generosity) go towards just not giving humans anything they need to survive, because they are useless compared to more efficient AI systems and instead have loyal AI servants do everything for them.
It’s basically the same problem as AIs that don’t value you having instrumental incentives to kill you and everyone else to take their resources.
Now to be clear, I think Vladimir Putin wouldn’t straight up kill/not give ethnic Russians what they need to survive, because for ethnic Russians, I’d expect he slightly cares about them, and slightly caring about people is enough to make people fantastically wealthy in the AI era, but 1) this is not something I would say for all world leaders and 2) the fact that AI empowered humans or AIs themselves only need to care about humans a little to prevent this scenario also makes AI alignment less important from a survival perspective, and means even methods that fail to precisely shape AI goals may be effective enough from a survival perspective, and also at least partially defuses the counting argument for misalignment being deadly, and in practice makes alignment relatively easy even for incompetent humans who don’t know what they are doing.
An issue here re the CEV of powerful people like Vladimir Putin not being that bad compared to nothingness is that currently, even dictators require most people in at least somewhat functional states in order to have power, and more generally the generosity of capitalism is at least in part based on humans being useful when fed and educated.
But in a world where AI automates away human labor, the incentives for powerful people (absent even mild values of compassion/generosity) go towards just not giving humans anything they need to survive, because they are useless compared to more efficient AI systems and instead have loyal AI servants do everything for them.
It’s basically the same problem as AIs that don’t value you having instrumental incentives to kill you and everyone else to take their resources.
The best piece on this subject is Defining The Intelligence Curse, with a link here (though Jan Bentley makes the point that since AIs can keep improving, you don’t actually want to be a rentier, and you instead just keep charging forward, leading to something like the ascended economy scenario described by Scott Alexander here.)
Now to be clear, I think Vladimir Putin wouldn’t straight up kill/not give ethnic Russians what they need to survive, because for ethnic Russians, I’d expect he slightly cares about them, and slightly caring about people is enough to make people fantastically wealthy in the AI era, but 1) this is not something I would say for all world leaders and 2) the fact that AI empowered humans or AIs themselves only need to care about humans a little to prevent this scenario also makes AI alignment less important from a survival perspective, and means even methods that fail to precisely shape AI goals may be effective enough from a survival perspective, and also at least partially defuses the counting argument for misalignment being deadly, and in practice makes alignment relatively easy even for incompetent humans who don’t know what they are doing.