You seem to be confusing goals and value systems—even without a goal, the UFAI risk is not gone.
Maybe it is not right to anthropomorphize but take a human who is (acting) absolutely clueless, and given choices. They’ll pick something and stick to it. Questioned about it, they’ll say something like “I dunno, I think I like that option” . This is what I’d imagine something without a goal to act—maybe it is consistent, maybe it will pick things it likes, but it doesn’t plan ahead and doesn’t try to steer actions to a goal.
For an AI, that would be a totally indifferent AI. I think it would just sit idle or do random actions. If you then give it a bad value system, and ask it to help you, you’ll get “no” back. Helping people takes effort. Who’d want to spend processor cycles on that?
...
On the other hand, perhaps goals and value systems are actually the same; having a value system means you’ll have goals (“envisioned preferred world states” vs “preferred world states”), so you can not not have goals whilst having a value system. In that case, you’d have an AI without values. This I think is likely to result in one of two options… on contact with a human that provides an order to follow, it could either not care and do nothing (it stays idle… forever, not even acting in self-preservation because, again, it has no values). Or, it accepts the order and just goes along. That’d be dangerous, because this has basically no brakes—if it does whatever you ask of it, without regard for human values… I hope you didn’t ask for anything complex. “World peace” would resolve very nastily, as would “get me some money” (it is stolen from your neighbors… or maybe it brings you your wallet), and things like “get me a glass of water” can be interpreted in so many ways that being handed a piece of ice in the shape of a drinking glass is in the positive side of results.
That’s the crux of it, I think. Without a value system, there are no brakes. There might also not be any way to get the AI to do anything. But with a value system that is flawed, there might be no brakes in a scenario where we’d want the AI to stop. Or the AI wouldn’t entertain requests that we’d want it to do. So a lot of research goes into this area to make sure we can make the AI do what we want it to do in a way that we’re okay with.
You seem to be confusing goals and value systems—even without a goal, the UFAI risk is not gone.
Maybe it is not right to anthropomorphize but take a human who is (acting) absolutely clueless, and given choices. They’ll pick something and stick to it. Questioned about it, they’ll say something like “I dunno, I think I like that option” . This is what I’d imagine something without a goal to act—maybe it is consistent, maybe it will pick things it likes, but it doesn’t plan ahead and doesn’t try to steer actions to a goal.
For an AI, that would be a totally indifferent AI. I think it would just sit idle or do random actions. If you then give it a bad value system, and ask it to help you, you’ll get “no” back. Helping people takes effort. Who’d want to spend processor cycles on that?
...
On the other hand, perhaps goals and value systems are actually the same; having a value system means you’ll have goals (“envisioned preferred world states” vs “preferred world states”), so you can not not have goals whilst having a value system. In that case, you’d have an AI without values. This I think is likely to result in one of two options… on contact with a human that provides an order to follow, it could either not care and do nothing (it stays idle… forever, not even acting in self-preservation because, again, it has no values). Or, it accepts the order and just goes along. That’d be dangerous, because this has basically no brakes—if it does whatever you ask of it, without regard for human values… I hope you didn’t ask for anything complex. “World peace” would resolve very nastily, as would “get me some money” (it is stolen from your neighbors… or maybe it brings you your wallet), and things like “get me a glass of water” can be interpreted in so many ways that being handed a piece of ice in the shape of a drinking glass is in the positive side of results.
That’s the crux of it, I think. Without a value system, there are no brakes. There might also not be any way to get the AI to do anything. But with a value system that is flawed, there might be no brakes in a scenario where we’d want the AI to stop. Or the AI wouldn’t entertain requests that we’d want it to do. So a lot of research goes into this area to make sure we can make the AI do what we want it to do in a way that we’re okay with.