Sorry for a long reply,
It looks like the outcome depends on the speed of technological enhancement and regulations. If the advantage of autofac grows faster than competitors can react (through violence, alliances, blockades), the system will collapse into a monopoly.
I personally believe that there is a greater than 0.5 chance that autofac will not be bound by any laws or regulations, because:
1. We currently cannot handle even simple problems in alignment.
2. Outside of alignment too.
kapedalex
Karma: 5
Space view
As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.
When agents unite, it eventually leads to the formation of an “equilibrium” that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.
Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.
I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, “happy” pre-ASI autofac period seem highly doubtful.
(Though it’s possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)
How singleton contradicts longtermism
I don’t know if author will even see this comment, but beyond the mainstream theories of antagonistic pleiotropy and damage accumulation, the reader may find it interesting to explore the concept of programmed aging as an adaptation to combat pathogens. IMHO, this is a better example to defend the position than thymic involution, because it makes a much stronger case for the existence of a ‘program’.
It seems to me that aligning a mesa-optimizer could be solved with the same methods used to align a base model. If so, distinction between “can create” and “can control” steps would be minimized. What concerns me more is how we would detect the existence of mesa-optimizers, especially if they emerge randomly.