This seems like an excellent argument to pause somewhere at about the AGI level, where a mistake is likely to give us a competent sentient computer virus and/or a new criminal organization and/or a nation of rather inconvenient competent people in a data-center: problems large enough to act as warning shots and learning exercises, but not actually wipe us out or permanently disempower us.
However, that would of course require:
a) actual willingness and capability to pause, globally (including China), and also b) correct judgement of whether the likely resulting problems will be dealable with rather than terminal (because they moved through the Ardennes faster than we had anticipated), and also c) an accurate idea on timelines of how far away this point is, and also d) that rogue AIs can’t get access to enough compute and research competence to self-improve up to levels where it is terminal.
So maybe we should in fact pause before that? Of course, then we don’t learn as much, or provide as loud a set of warning shots…
So maybe we should instead proceed very slowly and cautiously, rather than racing?
This seems like an excellent argument to pause somewhere at about the AGI level, where a mistake is likely to give us a competent sentient computer virus and/or a new criminal organization and/or a nation of rather inconvenient competent people in a data-center: problems large enough to act as warning shots and learning exercises, but not actually wipe us out or permanently disempower us.
However, that would of course require:
a) actual willingness and capability to pause, globally (including China), and also
b) correct judgement of whether the likely resulting problems will be dealable with rather than terminal (because they moved through the Ardennes faster than we had anticipated), and also
c) an accurate idea on timelines of how far away this point is, and also
d) that rogue AIs can’t get access to enough compute and research competence to self-improve up to levels where it is terminal.
So maybe we should in fact pause before that? Of course, then we don’t learn as much, or provide as loud a set of warning shots…
So maybe we should instead proceed very slowly and cautiously, rather than racing?