Rather flogs a dead horse, but highlights an important difference in perspective. You tell your AI to produce paperclips, and eventually it stops and asks if you would like it to do something different.
You could think “hey, cool, its actually doing friendly stuff I didn’t ask for”, or you could think “wait … how would knowing what I really want help it produce more paperclips … ”
Yeah. Though actually it’s more of a simplified version of a more serious problem.
One day you may give AI precise set of instructions, which you think would make good. Like find a way of curing diseases, but without harming patients, and without harming people for the sake of research and so on.
And you may find that your AI is perfectly friendly, but it wouldn’t yet mean it actually is. It may simply have learned human values as a mean of securing its existence and gaining power.
EDIT: And after gaining enough power it may as well help improve human health even more or reprogram human race to think unconditionaly that diseases were eradicated.
Rather flogs a dead horse, but highlights an important difference in perspective. You tell your AI to produce paperclips, and eventually it stops and asks if you would like it to do something different.
You could think “hey, cool, its actually doing friendly stuff I didn’t ask for”, or you could think “wait … how would knowing what I really want help it produce more paperclips … ”
Yeah. Though actually it’s more of a simplified version of a more serious problem.
One day you may give AI precise set of instructions, which you think would make good. Like find a way of curing diseases, but without harming patients, and without harming people for the sake of research and so on. And you may find that your AI is perfectly friendly, but it wouldn’t yet mean it actually is. It may simply have learned human values as a mean of securing its existence and gaining power.
EDIT: And after gaining enough power it may as well help improve human health even more or reprogram human race to think unconditionaly that diseases were eradicated.