[I]ntelligent machines will probably not be so literal-minded.
This is a variation of the “Superintelligent AI will do what you mean, not what you literally say; it would have to be pretty non-superintelligent to screw that up.”
The counter-argument is: The person making the request may not understand the full implications of “what they really mean”. The AI needs to be able to protect against bad unintended outcomes even of correctly interpreted requests. Because a superintelligent AI is very powerful the bad outcomes could be very bad indeed. To deal with this, the AI has to understand “what we really want”, which is tricky since most of the time we don’t even know what that is in any great detail.
This is a variation of the “Superintelligent AI will do what you mean, not what you literally say; it would have to be pretty non-superintelligent to screw that up.”
...except that my comments were fine, while the position that you are likening them to is completely daft. That doesn’t seem to be entirely fair. Maybe you thought I was making that daft argument—in which case, perhaps revisit the situation now that you have heard me state that I wasn’t.
This is a variation of the “Superintelligent AI will do what you mean, not what you literally say; it would have to be pretty non-superintelligent to screw that up.”
The counter-argument is: The person making the request may not understand the full implications of “what they really mean”. The AI needs to be able to protect against bad unintended outcomes even of correctly interpreted requests. Because a superintelligent AI is very powerful the bad outcomes could be very bad indeed. To deal with this, the AI has to understand “what we really want”, which is tricky since most of the time we don’t even know what that is in any great detail.
...except that my comments were fine, while the position that you are likening them to is completely daft. That doesn’t seem to be entirely fair. Maybe you thought I was making that daft argument—in which case, perhaps revisit the situation now that you have heard me state that I wasn’t.
I re-read your comment, but I’m still not sure what you’re driving at. Can you elaborate a little further?