Orthogonality: given those motivations, the AI will act to best implement them. This makes it “stupid” from an outside perspective that assumes it has conventional goals, but smart to those who know its strange motivation.
Not only that.
Preferring highly uncertain outcomes means you want to live in an unpredictable world. A world which you, the AI cannot predict. A simple way to get yourself into an unpredictable world is to make yourself dumb.
A simple way to get yourself into an unpredictable world is to make yourself dumb.
Er, no. At no point will the AI conclude “making my next iteration dumber will successfully make the world more unpredictable.” It will want the world to be more unpredictable, not the appearance of unpredictability to itself (which is just another form of wireheading—and to get a successful AI of any sort, we need to solve the wireheading issue).
I agree that it’s related to the wireheading problem. However unpredictability is a two-argument function and I wouldn’t be that confident about what an AI will or will not conclude.
Not only that.
Preferring highly uncertain outcomes means you want to live in an unpredictable world. A world which you, the AI cannot predict. A simple way to get yourself into an unpredictable world is to make yourself dumb.
Er, no. At no point will the AI conclude “making my next iteration dumber will successfully make the world more unpredictable.” It will want the world to be more unpredictable, not the appearance of unpredictability to itself (which is just another form of wireheading—and to get a successful AI of any sort, we need to solve the wireheading issue).
I agree that it’s related to the wireheading problem. However unpredictability is a two-argument function and I wouldn’t be that confident about what an AI will or will not conclude.
It’s estimate of objective factors in the world would not be impaired.