It seems to me that aligning a mesa-optimizer could be solved with the same methods used to align a base model. If so, distinction between “can create” and “can control” steps would be minimized. What concerns me more is how we would detect the existence of mesa-optimizers, especially if they emerge randomly.
kapedalex
Karma: 5
Cap-and-trade question: AI-2040
Space view
Sorry for a long reply,
It looks like the outcome depends on the speed of technological enhancement and regulations. If the advantage of autofac grows faster than competitors can react (through violence, alliances, blockades), the system will collapse into a monopoly.
I personally believe that there is a greater than 0.5 chance that autofac will not be bound by any laws or regulations, because:
1. We currently cannot handle even simple problems in alignment.
2. Outside of alignment too.
As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.
When agents unite, it eventually leads to the formation of an “equilibrium” that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.
Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.
I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, “happy” pre-ASI autofac period seem highly doubtful.
(Though it’s possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)
How singleton contradicts longtermism
I don’t know if author will even see this comment, but beyond the mainstream theories of antagonistic pleiotropy and damage accumulation, the reader may find it interesting to explore the concept of programmed aging as an adaptation to combat pathogens. IMHO, this is a better example to defend the position than thymic involution, because it makes a much stronger case for the existence of a ‘program’.
“Higher tax rates merely increase the incentive to avoid taxation, they don’t guarantee malfeasance.”
From my point of view these are the same thing in current scenario.
”It is very much possible that agents merely choose not to work, as the cost of marginal effort is not sufficiently rewarded with enough marginal income.”
As I mentioned: “It is reasonable to assume that the tax is too high when working on a project capable of making you the wealthiest agent requires a couple of data centers and a government license. Furthermore, one shouldn’t overlook the law of large numbers: among the many agents, there are bound to be not only those who are willing but also those who are capable.”
So I suppose this may hold true in other situations as well, but in the AGI race, your future earnings depend heavily on current work.
In general, it is better to eliminate the economic incentives to operate as a secret lab.