The posts you linked to are injunctive, not descriptive, and I note that humans are certainly prone to this kind of cheating, or else we wouldn’t have cocaine addicts (and yes, I’ve even seen very smart and rational people develop addictions, although they tend to be somewhat quicker to break them). It would depend on design, of course, but why should we expect that AIs would not also do this? Or, more to the point, how could we design them so that they don’t?
(On the other hand, maybe this is more a question about human neuropsychology than about AI design.)
People are only in favor of shortcuts in some areas—generally, where the “point” of that reward isn’t the person’s own goal. So, people will use contraceptives if they want to have sex for pleasure, even if the reward exists to make them reproduce (to anthropomorphize evolution). People might use drugs to feel okay because they are trying to feel okay, not accomplish goals by feeling okay (only an example). On the other hand, many (most?) people interested in, say, solving world hunger would reject a pill that gives them a false sense of having solved world hunger, because they’re trying to accomplish an external goal, rather than induce the mental state associated with accomplishing that goal. At least, all that’s according to my understanding at this point.
A properly designed AI would not modify its utility function to infinity, because that would not maximize its current utility function.
See Morality as Fixed Computation, The Domain of Your Utility Function, and Maximise Expected Utility, not Expected Perception of Utility.
The posts you linked to are injunctive, not descriptive, and I note that humans are certainly prone to this kind of cheating, or else we wouldn’t have cocaine addicts (and yes, I’ve even seen very smart and rational people develop addictions, although they tend to be somewhat quicker to break them). It would depend on design, of course, but why should we expect that AIs would not also do this? Or, more to the point, how could we design them so that they don’t?
(On the other hand, maybe this is more a question about human neuropsychology than about AI design.)
People are only in favor of shortcuts in some areas—generally, where the “point” of that reward isn’t the person’s own goal. So, people will use contraceptives if they want to have sex for pleasure, even if the reward exists to make them reproduce (to anthropomorphize evolution). People might use drugs to feel okay because they are trying to feel okay, not accomplish goals by feeling okay (only an example). On the other hand, many (most?) people interested in, say, solving world hunger would reject a pill that gives them a false sense of having solved world hunger, because they’re trying to accomplish an external goal, rather than induce the mental state associated with accomplishing that goal. At least, all that’s according to my understanding at this point.
Thanks: what I meant, several thousand times clearer!