(2) AIs assert full control … (2) is incoherent because the vast majority of humans would deeply prefer not to be permanently and totally disempowered … humanity will still want to keep doing all the things humanity likes doing; no one wants to be exterminated, put in a zoo, or otherwise prevented from living freely
That’s not what “incoherent” means. It’s a bad outcome, but it doesn’t follow that it therefore can’t actually happen, or that it can’t be clearly imagined as a real possibility. AIs that take control can enforce this state of affars as a permanent equilibrium even if the future of humanity isn’t literally extinguished. Humanity’s will for things to be different doesn’t have the power to ever break that equilibrium (if we end up there), doesn’t make it at all unstable.
The claim is that (2) is incoherent as a stable long-term equilibrium, not that it is logically impossible or can’t happen. I qualify the use of ‘coherent’ at the top of the piece, but I’ve clarified the language to make this more explicit, thanks for the correction.
Edit, responding to your edit: whether (2) is permanently enforceable or not, the core conclusion is the same AFAICT. If it’s unstable, then we end up needing (3) anyway; if it’s stable, then the window for building toward (3) is finite and we’re basically wasting it. Either way the implication is to do the work now.
My point is, it’s not incoherent as a stable long-term equilibrium. There’s nothing unstable about that equilibrium in the long term, literally indefinitely. The arguments from humanity’s desires don’t work, because there’s nothing in that equilibrium that converts those desires into outcomes, nothing to give those desires the power to ever destabilize the equilibrium.
Took the note, updated the intro. (2) can be stable and still be definitely-worth-avoiding, which is all the argument I am making in this piece requires.
The argument for (1) and (2) being unstable seems to fail on empirical grounds. Consider the current situation between humans and domesticated chickens. Humans have full control. Presumably, if the chickens have feelings, they are quite unhappy about this. Yet there is nothing they can do to change it. The equilibrium is stable (absent intervention from e.g. chicken-loving humans).
Maybe you mean something different at the beginning of your post, but idk what that might be.
That’s not what “incoherent” means. It’s a bad outcome, but it doesn’t follow that it therefore can’t actually happen, or that it can’t be clearly imagined as a real possibility. AIs that take control can enforce this state of affars as a permanent equilibrium even if the future of humanity isn’t literally extinguished. Humanity’s will for things to be different doesn’t have the power to ever break that equilibrium (if we end up there), doesn’t make it at all unstable.
The claim is that (2) is incoherent as a stable long-term equilibrium, not that it is logically impossible or can’t happen. I qualify the use of ‘coherent’ at the top of the piece, but I’ve clarified the language to make this more explicit, thanks for the correction.
Edit, responding to your edit: whether (2) is permanently enforceable or not, the core conclusion is the same AFAICT. If it’s unstable, then we end up needing (3) anyway; if it’s stable, then the window for building toward (3) is finite and we’re basically wasting it. Either way the implication is to do the work now.
My point is, it’s not incoherent as a stable long-term equilibrium. There’s nothing unstable about that equilibrium in the long term, literally indefinitely. The arguments from humanity’s desires don’t work, because there’s nothing in that equilibrium that converts those desires into outcomes, nothing to give those desires the power to ever destabilize the equilibrium.
Took the note, updated the intro. (2) can be stable and still be definitely-worth-avoiding, which is all the argument I am making in this piece requires.
If it can be stable, it’s clearly not incoherent. You are making knowably invalid (local) claims to justify your overall message.
The argument for (1) and (2) being unstable seems to fail on empirical grounds. Consider the current situation between humans and domesticated chickens. Humans have full control. Presumably, if the chickens have feelings, they are quite unhappy about this. Yet there is nothing they can do to change it. The equilibrium is stable (absent intervention from e.g. chicken-loving humans).
Maybe you mean something different at the beginning of your post, but idk what that might be.