If they started working at smiley-face pin factories, that would be because they predicted that that would maximize something. If that something is number of smiles, they wouldn’t work at the factory because it would cause you to shut them off. They would act such that you think they are Friendly until you are powerless to stop them.
We might be dealing with the sort of utility-maximizing loophole that doesn’t occur to an AI until it is intelligent enough to keep quiet. If your dog were made happy by smiles, he wouldn’t try to start a factory, but he would do things that made you smile in the past, and you might be tempted to increase his intelligence to help him in his efforts.
If they started working at smiley-face pin factories, that would be because they predicted that that would maximize something. If that something is number of smiles, they wouldn’t work at the factory because it would cause you to shut them off. They would act such that you think they are Friendly until you are powerless to stop them.
If the first AIs are chimp-smart, though(or dog-smart, or dumber), they won’t be capable of thinking that far ahead.
We might be dealing with the sort of utility-maximizing loophole that doesn’t occur to an AI until it is intelligent enough to keep quiet. If your dog were made happy by smiles, he wouldn’t try to start a factory, but he would do things that made you smile in the past, and you might be tempted to increase his intelligence to help him in his efforts.