Have this alignment and the surrounding dynamics cause humans to choose to remain in control over time, or somehow be unable to choose differently.
This is self-contradictory: if the surrounding dynamics strongly preclude humans from “choosing otherwise”, humans are no longer “in control”. Also, under certain definitions of “choosing differently”, humans may be precluded from moving into different biological and computational substrates, which in itself might be a cosmic tragedy because it may forever preclude humans from realising vast amounts of potential.
It is not clear to what extent robust alignment is a coherent concept especially in a competitive world or even how it interacts with maximization, as it contains many potential contradictions and requirements.
This is self-contradictory: if the surrounding dynamics strongly preclude humans from “choosing otherwise”, humans are no longer “in control”. Also, under certain definitions of “choosing differently”, humans may be precluded from moving into different biological and computational substrates, which in itself might be a cosmic tragedy because it may forever preclude humans from realising vast amounts of potential.
And Zvi points out these contradictions himself: