If this is leading in a direction where “wireheading” is identified with “being happy and living a fulfilled life”, then we might as well head it off at the pass.
Being happy—being in a pleasurable state—isn’t enough, we would insist that our future lives should also be meaningful (which I would argue is part of “fulfilled”).
This isn’t merely a subjective attribute, as is “happy” which could be satisfied by permanently blissing out. It has objective consequences; you can tell “meaningful” from the outside. Meaningful arrangements of matter are improbable but lawful, structured but hard to predict, and so on.
“Being totally happy all the time” is a state of mind, the full description of which would compress very well, just as the description of zillions of molecules of gas can be compressed to a handful of parameters. “Meaningful” corresponds to states of mind with more structure and order.
If we are to be somehow “fixed” we would want the “fix” to preserve or restore the property we have now, of being the type of creature who can (and in fact do) choose for themselves.
The preference for “objective meaningfulness”—for states which do not compress very well—seems to me a fairly arbitrary (meaningless) preference. I don’t think it’s much different from paperclip maximization.
Who is to observe the “meaningful” states, if everyone is in a state where they are happy?
I am not even convinced that “happy and fulfilled” compresses easily. But if it did, what is the issue? Everyone will be so happy as to not mind the absence of complicated states.
I would go so far as to say that seeking complicated states is something we do right now because it is the most engaging substitute we have for being happy.
And not everyone does this. Most people prefer to empty their minds instead. It may even be that seeking complexity is a type of neurotic tendency.
Should the FAI be designed with a neurotic tendency?
If this is leading in a direction where “wireheading” is identified with “being happy and living a fulfilled life”, then we might as well head it off at the pass.
Being happy—being in a pleasurable state—isn’t enough, we would insist that our future lives should also be meaningful (which I would argue is part of “fulfilled”).
This isn’t merely a subjective attribute, as is “happy” which could be satisfied by permanently blissing out. It has objective consequences; you can tell “meaningful” from the outside. Meaningful arrangements of matter are improbable but lawful, structured but hard to predict, and so on.
“Being totally happy all the time” is a state of mind, the full description of which would compress very well, just as the description of zillions of molecules of gas can be compressed to a handful of parameters. “Meaningful” corresponds to states of mind with more structure and order.
If we are to be somehow “fixed” we would want the “fix” to preserve or restore the property we have now, of being the type of creature who can (and in fact do) choose for themselves.
The preference for “objective meaningfulness”—for states which do not compress very well—seems to me a fairly arbitrary (meaningless) preference. I don’t think it’s much different from paperclip maximization.
Who is to observe the “meaningful” states, if everyone is in a state where they are happy?
I am not even convinced that “happy and fulfilled” compresses easily. But if it did, what is the issue? Everyone will be so happy as to not mind the absence of complicated states.
I would go so far as to say that seeking complicated states is something we do right now because it is the most engaging substitute we have for being happy.
And not everyone does this. Most people prefer to empty their minds instead. It may even be that seeking complexity is a type of neurotic tendency.
Should the FAI be designed with a neurotic tendency?
I’m not so sure.