I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, “happy” pre-ASI autofac period seem highly doubtful.
(Though it’s possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, “happy” pre-ASI autofac period seem highly doubtful.
It’s unclear to me that such a decisive resource advantage would be possible to hold unless we start from an already decisively unipolar world. So long as there are powers who can credibly threaten total dominance, there will be strategic reasons for state and business actors to prevent complete consolidation, and if desperate enough would use destructive force (or the threat of such force) to ensure one actor does not become dominant.
As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.
When agents unite, it eventually leads to the formation of an “equilibrium” that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.
Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.
I think I’m confused what your position is then. It’s true that economic competition is generally unstable in that the specific balance of power between competitors is unstable, but competition itself is often quite stable and doesn’t collapse into either chaos or monopoly unless there’s something weird about the competitive environment that allows this to happen (e.g. no rule of law, protective regulations, etc.).
I also expect the whole Autofac Era to feel quite unstable to people because things will be changing quickly. And I also don’t expect it to last too long, because I think it’s a short period of a few years between its start and the development of AGI (or, if for some reason AGI is impossible, then Hansonian EMs).
I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, “happy” pre-ASI autofac period seem highly doubtful.
(Though it’s possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)
It’s unclear to me that such a decisive resource advantage would be possible to hold unless we start from an already decisively unipolar world. So long as there are powers who can credibly threaten total dominance, there will be strategic reasons for state and business actors to prevent complete consolidation, and if desperate enough would use destructive force (or the threat of such force) to ensure one actor does not become dominant.
As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.
When agents unite, it eventually leads to the formation of an “equilibrium” that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.
Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.
I think I’m confused what your position is then. It’s true that economic competition is generally unstable in that the specific balance of power between competitors is unstable, but competition itself is often quite stable and doesn’t collapse into either chaos or monopoly unless there’s something weird about the competitive environment that allows this to happen (e.g. no rule of law, protective regulations, etc.).
I also expect the whole Autofac Era to feel quite unstable to people because things will be changing quickly. And I also don’t expect it to last too long, because I think it’s a short period of a few years between its start and the development of AGI (or, if for some reason AGI is impossible, then Hansonian EMs).