“My preferences” seem to be confused statement. There is how “me” looks from outside and how I look at myself from inside, but those are two different thing unless my conscious self is all that is to “me”. Any algorithm trying to find my true preferences have to analyze my brain, find conscious part(s?!) and somehow understand it. Anything else runs the risk of imprisoning me in my own body, mute and helpless.
Not easy to figure this out for aAlice if she doesn’t have a related concept but doable—at least if this distinction is something that is inferrable. I think it is as in manifests in exactly the differences between public and private speech and people do talk about consciousness.
“My preferences” seem to be confused statement. There is how “me” looks from outside and how I look at myself from inside, but those are two different thing unless my conscious self is all that is to “me”. Any algorithm trying to find my true preferences have to analyze my brain, find conscious part(s?!) and somehow understand it. Anything else runs the risk of imprisoning me in my own body, mute and helpless.
Not easy to figure this out for aAlice if she doesn’t have a related concept but doable—at least if this distinction is something that is inferrable. I think it is as in manifests in exactly the differences between public and private speech and people do talk about consciousness.