AI-doomers often suggest that their fears arise from special technical calculations. But in fact, their main argument is just the mere logical possibility of a huge sudden AI breakthrough, combined with a suddenly murderous AI inclination. [...]
However, AI-doomers insist on the logical possibility that such expectations could be wrong. An AI might suddenly and without warning explode in abilities, and just as fast change its priorities to become murderously indifferent to us.
I feel like both of these could be interpreted less as “strawman” and more as “imprecise wording”. E.g. “an AI might suddenly explode in abilities and change its priorities” sounds like it could be a description of a sharp left turn, where the AI starts acting hostile to humans after undergoing an increase in capabilities. Hanson has a habit of compressing complicated arguments to just a few sentences, which necessarily causes leaving out nuances like “the change in priorities is not due to self-modification of intrinsic goals but due to increased capabilities outstripping its alignment properties and leading to changed instrumental goals”.
I feel like both of these could be interpreted less as “strawman” and more as “imprecise wording”. E.g. “an AI might suddenly explode in abilities and change its priorities” sounds like it could be a description of a sharp left turn, where the AI starts acting hostile to humans after undergoing an increase in capabilities. Hanson has a habit of compressing complicated arguments to just a few sentences, which necessarily causes leaving out nuances like “the change in priorities is not due to self-modification of intrinsic goals but due to increased capabilities outstripping its alignment properties and leading to changed instrumental goals”.