Yeah, that’s my primary model here. I’d be surprised but not shocked if competition between countries explained most of the effect.
It seems worth noting here that when it looked for a while like the planned economy of the Soviet Union might outperform western free market economies (and even before that, when many intellectuals just thought based on theory that central planning would perform better *), there were a lot of people in the west who supported switching to socialism / central planning. Direct military competition (which Carl’s paper focuses on more) would make this pressure even stronger. So if one country switches to the “one AGI controls everything” model (either deliberately or due to weak/absent existing institutions that work against centralization), it seems hard for other countries to hold out in the long run.
Does that seem right to you, or do you see things turn out a different way (in the long run)?
(* I realize this is also a cautionary tale about using theory to predict the future, like I’m trying to do now.)
Does that seem right to you, or do you see things turn out a different way (in the long run)?
I agree that direct military competition would create such a pressure.
I’m not sure that absent that there actually is competition between countries—what are they even competing on? You’re reasoning as though they compete on economic efficiency, but what causes countries with lower economic efficiency to vanish? Perhaps in countries with lower economic efficiency, voters tend to put in a new government—but in that case it seems like really the competition between countries is on “what pleases voters”, which may not be exactly what we want but it probably isn’t too risky if we have an AGI-fueled government that’s intent-aligned with “what pleases voters”.
(It’s possible that you get politicians who look like they’re trying to please voters but once they have enough power they then serve their own interests, but this looks like “the government gains power, and the people no longer have effective control over government”.)
I’m not sure that absent that there actually is competition between countries—what are they even competing on? You’re reasoning as though they compete on economic efficiency, but what causes countries with lower economic efficiency to vanish?
I guess ultimately they’re competing to colonize the universe, or be one of the world powers that have some say in the fate of the universe? Absent military conflict, the less efficient countries won’t disappear, but they’ll fall increasingly behind in control of resources and overall bargaining power, and their opinions just won’t be reflected much in how the universe turns out.
In that case this model would only hold if governments:
Actually think through the long-term implications of AI
Think about this particular argument
Have enough certainty in this argument to actually act upon it
Notably, there aren’t any feedback loops for the thing-being-competed-on, and so natural-selection style optimization doesn’t happen. This makes me much less likely to believe in arguments of the form “The thing-being-competed-on will have a high value, because there is competition”—the mechanism that usually makes that true is natural selection or some equivalent.
I think I oversimplified my model there. Actually competing to colonize/influence the universe will be the last stage, when the long-term implications of AI and of this particular argument will already be clear. Before that, the dynamics would be driven more by things like internal political and economic processes (some countries already have authoritarian governments and would naturally gravitate towards more centralization of power through political means, and others do not have strong laws/institutions to prevent centralization of the economy through market forces), competition for power (such as diplomatic and military power) and prestige (both of which are desired by leaders and voters alike) on the world stage, and direct military conflicts.
All of these forces create pressure towards greater AGI-based centralization, while the only thing pushing against it appears to be political pressure in some countries against centralization of power. If those countries succeed in defending against centralization but fall significantly behind in economic growth as a result, they will end up not influencing the future of the universe much so we might as well ignore them and focus on the others.
It seems worth noting here that when it looked for a while like the planned economy of the Soviet Union might outperform western free market economies (and even before that, when many intellectuals just thought based on theory that central planning would perform better *), there were a lot of people in the west who supported switching to socialism / central planning. Direct military competition (which Carl’s paper focuses on more) would make this pressure even stronger. So if one country switches to the “one AGI controls everything” model (either deliberately or due to weak/absent existing institutions that work against centralization), it seems hard for other countries to hold out in the long run.
Does that seem right to you, or do you see things turn out a different way (in the long run)?
(* I realize this is also a cautionary tale about using theory to predict the future, like I’m trying to do now.)
I agree that direct military competition would create such a pressure.
I’m not sure that absent that there actually is competition between countries—what are they even competing on? You’re reasoning as though they compete on economic efficiency, but what causes countries with lower economic efficiency to vanish? Perhaps in countries with lower economic efficiency, voters tend to put in a new government—but in that case it seems like really the competition between countries is on “what pleases voters”, which may not be exactly what we want but it probably isn’t too risky if we have an AGI-fueled government that’s intent-aligned with “what pleases voters”.
(It’s possible that you get politicians who look like they’re trying to please voters but once they have enough power they then serve their own interests, but this looks like “the government gains power, and the people no longer have effective control over government”.)
I guess ultimately they’re competing to colonize the universe, or be one of the world powers that have some say in the fate of the universe? Absent military conflict, the less efficient countries won’t disappear, but they’ll fall increasingly behind in control of resources and overall bargaining power, and their opinions just won’t be reflected much in how the universe turns out.
In that case this model would only hold if governments:
Actually think through the long-term implications of AI
Think about this particular argument
Have enough certainty in this argument to actually act upon it
Notably, there aren’t any feedback loops for the thing-being-competed-on, and so natural-selection style optimization doesn’t happen. This makes me much less likely to believe in arguments of the form “The thing-being-competed-on will have a high value, because there is competition”—the mechanism that usually makes that true is natural selection or some equivalent.
I think I oversimplified my model there. Actually competing to colonize/influence the universe will be the last stage, when the long-term implications of AI and of this particular argument will already be clear. Before that, the dynamics would be driven more by things like internal political and economic processes (some countries already have authoritarian governments and would naturally gravitate towards more centralization of power through political means, and others do not have strong laws/institutions to prevent centralization of the economy through market forces), competition for power (such as diplomatic and military power) and prestige (both of which are desired by leaders and voters alike) on the world stage, and direct military conflicts.
All of these forces create pressure towards greater AGI-based centralization, while the only thing pushing against it appears to be political pressure in some countries against centralization of power. If those countries succeed in defending against centralization but fall significantly behind in economic growth as a result, they will end up not influencing the future of the universe much so we might as well ignore them and focus on the others.