The obvious counterargument to me is: Trying to get diffuse power distribution to work in this context also involves inventing a new thing from scratch on the first try.
I take the “diffuse power distribution” vision to be: We are going to transition from the current era where humanity is the dominant power (or where all great powers are groups of humans) to an era where the great powers are AIs and humans are lesser powers which are still useful for great powers to have as allies, and then to an era where humans are basically irrelevant to the great AI powers. And during this current era of human power, we are going to set up the balance & terms of power in ways that will get things to play out in a way that is favorable to humanity / human values even during these later eras where humans are increasingly disempowered.
My reaction to that is not we’ve seen things like this work, this is the one solution that doesn’t require inventing a new thing from scratch on the first try.
The obvious counterargument to me is: Trying to get diffuse power distribution to work in this context also involves inventing a new thing from scratch on the first try.
I take the “diffuse power distribution” vision to be: We are going to transition from the current era where humanity is the dominant power (or where all great powers are groups of humans) to an era where the great powers are AIs and humans are lesser powers which are still useful for great powers to have as allies, and then to an era where humans are basically irrelevant to the great AI powers. And during this current era of human power, we are going to set up the balance & terms of power in ways that will get things to play out in a way that is favorable to humanity / human values even during these later eras where humans are increasingly disempowered.
My reaction to that is not we’ve seen things like this work, this is the one solution that doesn’t require inventing a new thing from scratch on the first try.