We should expect ASI to indeed be radically transformative, to the point where our usual human abstractions like “multipolarity” or “authoritarianism” have very little meaning. This intuition caused me to dismiss most power-concentration threat models as sort of confused, or at least overconfident in their descriptions of future dynamics.
One objection, though, is that power-seeking humans may have an incentive to restrain the transformative properties of AI at a level that preserves human-shaped things like polarity or authoritarianism, given they benefit. So we shouldn’t be too surprised if we end up in a hellish power-concentrated world (absent loss of control).
I’ve always interpreted power concentration to be adjacent to loss of control, as only AI decision-making autonomy differs between them. I.e. AIs in LoC AIs are just a bit more confident when selecting goals compared to PC AIs, as they defer to humans in PC but don’t in LoC.
So, do you think PC isn’t a realistic threat model because you think it can easily devolve into LoC? Or do you think PC necessitates an oxymoronic AI that both has ultimate decision-making ability (as it must if it were to accomplish the millions of subtasks concentrating power necessitates) and ultimate deference (remains loyal to a much less intelligent being. Or your qualm with PC something else?
We should expect ASI to indeed be radically transformative, to the point where our usual human abstractions like “multipolarity” or “authoritarianism” have very little meaning. This intuition caused me to dismiss most power-concentration threat models as sort of confused, or at least overconfident in their descriptions of future dynamics.
One objection, though, is that power-seeking humans may have an incentive to restrain the transformative properties of AI at a level that preserves human-shaped things like polarity or authoritarianism, given they benefit. So we shouldn’t be too surprised if we end up in a hellish power-concentrated world (absent loss of control).
I’ve always interpreted power concentration to be adjacent to loss of control, as only AI decision-making autonomy differs between them. I.e. AIs in LoC AIs are just a bit more confident when selecting goals compared to PC AIs, as they defer to humans in PC but don’t in LoC.
So, do you think PC isn’t a realistic threat model because you think it can easily devolve into LoC? Or do you think PC necessitates an oxymoronic AI that both has ultimate decision-making ability (as it must if it were to accomplish the millions of subtasks concentrating power necessitates) and ultimate deference (remains loyal to a much less intelligent being. Or your qualm with PC something else?