Your post does show that trying to maintain control over a superpoweful AI ends in disaster with high probability.
We do know that there are efforts which disclaim maintaining control over their AIs (which presumably involves different risks, but probably not the risks described in this post, at least if each AI in question is sufficiently distributed geographically, rather than locally concentrated).
Do you assume that those efforts are doomed to lose to efforts based on control in terms of speed of technical progress and therefore can be disregarded, or do you mean to analyze that class of efforts and their safety problems elsewhere?
You will build ASI first and then establish an eternal utopia, right?
Note that one needs control over the leading ASI if one wants to become a dictator, but if one is actually aiming for utopia, then control-based approaches are likely to be highly counter-productive. Humans don’t have good track record in their utopia-building attempts, to say the least.
EDIT: I am aware of at least one project whose leader is disclaiming control and pushing for a different approach (Ben Goertzel) and of at least one project whose leader has a history of being very skeptical of the control approach and of pushing for different approaches not involving long-term human control over AI (Ilya Sutskever). It’s likely that there are more of those. With Ben, it’s difficult to say if his org has a chance, but they are specifically pushing for a very distributed architecture, which is not easy to fork or to take over.
Your post does show that trying to maintain control over a superpoweful AI ends in disaster with high probability.
We do know that there are efforts which disclaim maintaining control over their AIs (which presumably involves different risks, but probably not the risks described in this post, at least if each AI in question is sufficiently distributed geographically, rather than locally concentrated).
Do you assume that those efforts are doomed to lose to efforts based on control in terms of speed of technical progress and therefore can be disregarded, or do you mean to analyze that class of efforts and their safety problems elsewhere?
Note that one needs control over the leading ASI if one wants to become a dictator, but if one is actually aiming for utopia, then control-based approaches are likely to be highly counter-productive. Humans don’t have good track record in their utopia-building attempts, to say the least.
EDIT: I am aware of at least one project whose leader is disclaiming control and pushing for a different approach (Ben Goertzel) and of at least one project whose leader has a history of being very skeptical of the control approach and of pushing for different approaches not involving long-term human control over AI (Ilya Sutskever). It’s likely that there are more of those. With Ben, it’s difficult to say if his org has a chance, but they are specifically pushing for a very distributed architecture, which is not easy to fork or to take over.