Cogent framing; thanks for writing it.
I’d be very interested to read your framing for the problem of “how do we get to a good future for humanity, conditional on the first attractor state for AGI alignment?”[1]
Here is a brainstorm of the big problems that remain once we successfully get into the first attractor state:
Concentration of power / power grab risk. Liberal democracy does not work by preventing terrible people from getting into positions of power; it works by spreading out the power in a system of checks and balances and red tape transparency (free press, free speech) and term limits, that functions to limit what the terrible people can do in power. Once we get to ASI, the ASI project will determine the course of the future, not the traditional government+press system. (Because the ASI project will be able to easily manipulate those traditional institutions if it wants to) So somehow we need to design the governance structure of the ASI project to have similar checks and balances etc. as liberal democracy—because by default the governance structure of the ASI project will be akin to an authoritarian dictatorship, just like most companies are and just like the executive branch (considered in isolation) is. Otherwise, we are basically crossing our fingers and hoping that the men in charge of the project will be humble, cosmopolitan, benevolent, etc. and devolve power instead of abusing it.
S-risk. This is related to the above but distinct from it. I’m quite worried about this actually.
...actually everything else is a distant second as far as I can tell (terrorist misuse, China winning, wealth inequality, philosophical mistakes… or a distant distant third (wealth inequality, unemployment, meaning))
Cogent framing; thanks for writing it. I’d be very interested to read your framing for the problem of “how do we get to a good future for humanity, conditional on the first attractor state for AGI alignment?”[1]
Would you frame it as “the AGI lab leadership alignment problem”? Or a governance problem? Or something else?
Here is a brainstorm of the big problems that remain once we successfully get into the first attractor state:
Concentration of power / power grab risk. Liberal democracy does not work by preventing terrible people from getting into positions of power; it works by spreading out the power in a system of checks and balances and red tape transparency (free press, free speech) and term limits, that functions to limit what the terrible people can do in power. Once we get to ASI, the ASI project will determine the course of the future, not the traditional government+press system. (Because the ASI project will be able to easily manipulate those traditional institutions if it wants to) So somehow we need to design the governance structure of the ASI project to have similar checks and balances etc. as liberal democracy—because by default the governance structure of the ASI project will be akin to an authoritarian dictatorship, just like most companies are and just like the executive branch (considered in isolation) is. Otherwise, we are basically crossing our fingers and hoping that the men in charge of the project will be humble, cosmopolitan, benevolent, etc. and devolve power instead of abusing it.
S-risk. This is related to the above but distinct from it. I’m quite worried about this actually.
...actually everything else is a distant second as far as I can tell (terrorist misuse, China winning, wealth inequality, philosophical mistakes… or a distant distant third (wealth inequality, unemployment, meaning))
Could you say more about how you think S-risks could arise from the first attractor state?