It’s interesting to look at people’s arguments against paperclip maximizers. There seem to be two related categories that make up most of the objections:
People who can’t imagine that a sufficiently intelligent being could be that different from us. One guy tried to claim that morality is universal so of course an artificial intelligence would share our values. Another said that a superintelligence would inevitably realize that its existence was pointless, as if there could possibly be some point other than maximizing the number of paperclips. Another claimed that morality is an “emergent phenomenon”, but didn’t explain what that actually meant, or how humanlike morality would emerge from a being whose only goal is paperclip maximization.
People who think of it as a dumb machine, more akin to a drill press than an alien god. Just put an off button on it! Or require that there be a human operator with an axe ready to cut the power lines to the computer.
What these objections both have in common is that they assume that the world consists of humanlike intelligences or dumb machines. It’s unintuitive to imagine something that is both intelligent and profoundly alien.
I’d split the difference—I don’t think it’s that hard to imagine an AI which has about as much loyalty to Ais as people have to people.
Really alien minds are naturally much harder to imagine. Clippy seems more like a damaged human than a thoroughly alien mind.
This may be a matter of assuming that minds would naturally have a complex mix of entangled goals, the way humans do. Even an FAI has two goals (Friendliness and increasing its intelligence) which may come into conflict.
Faint memory: an Alexis Gilliland cartoon of an automated bomber redirecting its target from a robot factory to a maternity ward.
Even an FAI has two goals (Friendliness and increasing its intelligence) which may come into conflict.
No, just Friendliness. Increasing intelligence has no weight whatsoever as a terminal goal. Of course, an AI that did not increase its intelligence to a level which it could do anything practical to aid me (or whatever the AI is Friendly to) is trivially not Friendly a posteriori.
A discussion of paperclip maximizers (linking here) has made the front page of reddit.
It’s interesting to look at people’s arguments against paperclip maximizers. There seem to be two related categories that make up most of the objections:
People who can’t imagine that a sufficiently intelligent being could be that different from us. One guy tried to claim that morality is universal so of course an artificial intelligence would share our values. Another said that a superintelligence would inevitably realize that its existence was pointless, as if there could possibly be some point other than maximizing the number of paperclips. Another claimed that morality is an “emergent phenomenon”, but didn’t explain what that actually meant, or how humanlike morality would emerge from a being whose only goal is paperclip maximization.
People who think of it as a dumb machine, more akin to a drill press than an alien god. Just put an off button on it! Or require that there be a human operator with an axe ready to cut the power lines to the computer.
What these objections both have in common is that they assume that the world consists of humanlike intelligences or dumb machines. It’s unintuitive to imagine something that is both intelligent and profoundly alien.
I’d split the difference—I don’t think it’s that hard to imagine an AI which has about as much loyalty to Ais as people have to people.
Really alien minds are naturally much harder to imagine. Clippy seems more like a damaged human than a thoroughly alien mind.
This may be a matter of assuming that minds would naturally have a complex mix of entangled goals, the way humans do. Even an FAI has two goals (Friendliness and increasing its intelligence) which may come into conflict.
Faint memory: an Alexis Gilliland cartoon of an automated bomber redirecting its target from a robot factory to a maternity ward.
No, just Friendliness. Increasing intelligence has no weight whatsoever as a terminal goal. Of course, an AI that did not increase its intelligence to a level which it could do anything practical to aid me (or whatever the AI is Friendly to) is trivially not Friendly a posteriori.
That leads to an interesting question—how would an FAI decide how much intelligence is enough?
I don’t know. It’s supposed to be the smart one, not me. ;)
I’m hoping it goes something like:
Predict the expected outcome of choosing to self improve some more.
Predict the expected outcome of choosing not to self improve some more.
Do the one that gives the best probability distribution of expected results.