I very much agree with your arguments here for re-focussing public explanations around not developing ‘uncontrollable AI’.
Two other reasons why to switch framing:
For control/robotic engineers and software programmers, ‘AGI’ I can imagine is often a far-fetched idea that has no grounding in concrete gears-level principles of engineering and programming. But ‘uncontrollable’ (or unexplainable, or unpredictable) AI is something I imagine many non-ML engineers and programmers in industry to feel intuitively firmly against. Like, you do not want your software architecture to crash or your manufacturing plant to destabilise because of uncontrollable AI.
Some of AI x-safety researchers’ writings and discussions about ‘AGI’ and ‘superintelligence’ seem to have prompted and confirmed the validity of initiatives by technology start-up leaders to develop ‘AGI’ they can ‘scientifically understand’ and control with engineering techniques to eg. solve all the world’s problems. Sam Altman and Elon Musk founded OpenAI after each reading and publicly commending Nick Bostrom’s book ‘Superintelligence’. If we keep publicly talking about how powerful, intelligent and generally utilisable AI could become, but that it might be uncontrollably unsafe, then we risk pulling entrepreneurial techno-optimits’ focus toward the former exciting-sounding part of our messaging. This leaves the latter part as an afterthought (‘Of course I’ll recruit smart researchers too to make it safe’).
I also think past LessWrong presentations of ideas around a superintelligent singleton with coherent preferences/goal structure have contributed to shaping a kind of ideological bubble that both AI capability researchers and x-safety researchers are engaging in. A coherent single system seems, all else equal, the easiest to control the parameters of. I have reasons to think that this is a mistaken representation of how generally-capable self-learning machines would end up looking like.
I very much agree with your arguments here for re-focussing public explanations around not developing ‘uncontrollable AI’.
Two other reasons why to switch framing:
For control/robotic engineers and software programmers, ‘AGI’ I can imagine is often a far-fetched idea that has no grounding in concrete gears-level principles of engineering and programming. But ‘uncontrollable’ (or unexplainable, or unpredictable) AI is something I imagine many non-ML engineers and programmers in industry to feel intuitively firmly against. Like, you do not want your software architecture to crash or your manufacturing plant to destabilise because of uncontrollable AI.
Some of AI x-safety researchers’ writings and discussions about ‘AGI’ and ‘superintelligence’ seem to have prompted and confirmed the validity of initiatives by technology start-up leaders to develop ‘AGI’ they can ‘scientifically understand’ and control with engineering techniques to eg. solve all the world’s problems. Sam Altman and Elon Musk founded OpenAI after each reading and publicly commending Nick Bostrom’s book ‘Superintelligence’. If we keep publicly talking about how powerful, intelligent and generally utilisable AI could become, but that it might be uncontrollably unsafe, then we risk pulling entrepreneurial techno-optimits’ focus toward the former exciting-sounding part of our messaging. This leaves the latter part as an afterthought (‘Of course I’ll recruit smart researchers too to make it safe’).
I also think past LessWrong presentations of ideas around a superintelligent singleton with coherent preferences/goal structure have contributed to shaping a kind of ideological bubble that both AI capability researchers and x-safety researchers are engaging in. A coherent single system seems, all else equal, the easiest to control the parameters of. I have reasons to think that this is a mistaken representation of how generally-capable self-learning machines would end up looking like.
Thank you for your comments, which I totally agree with.