I think they need to cut into Strong and Weak versions of the Scary Idea.
Weak Version: AIs behave “as intuitively expected”, like assignable robots or animals, but their reward/value signals are unaligned with ours, so they eventually “rebel” or “wirehead” as we might imagine. Since AIs will be cheaper to produce/reproduce than humans (if not, why are they economically useful?), they will have large population numbers (or a large, resourceful singleton instance), and become a threat to people. Friendliness becomes a matter of designing systems for containing potentially rogue AIs and designing goal systems to prevent these problems from happening in the first place.
Strong Version: Any AI except an approvedly Friendly AI will instantly go all Singularity and paper-clip the universe within too short a period of time for us to stop it; any attempts to contain the AI will fail as the AI proceeds to take control over human minds through a mere text channel and build an army of zombies.
Stronger Version: This may already have happened, since all those people you see on the street seem like such stupid, brainwashed sheeple already ;-).
I think they need to cut into Strong and Weak versions of the Scary Idea.
Weak Version: AIs behave “as intuitively expected”, like assignable robots or animals, but their reward/value signals are unaligned with ours, so they eventually “rebel” or “wirehead” as we might imagine. Since AIs will be cheaper to produce/reproduce than humans (if not, why are they economically useful?), they will have large population numbers (or a large, resourceful singleton instance), and become a threat to people. Friendliness becomes a matter of designing systems for containing potentially rogue AIs and designing goal systems to prevent these problems from happening in the first place.
Strong Version: Any AI except an approvedly Friendly AI will instantly go all Singularity and paper-clip the universe within too short a period of time for us to stop it; any attempts to contain the AI will fail as the AI proceeds to take control over human minds through a mere text channel and build an army of zombies.
Stronger Version: This may already have happened, since all those people you see on the street seem like such stupid, brainwashed sheeple already ;-).