One reason I might not want to become a ball of ecstasy is that I don’t really feel it’s ME. I’m not even sure it’s sentient, since it doesn’t learn, communicate, or reason, it just enjoys.
Yes, but for example in highway hypnosis people drive on ‘boring’ stretches of highway and then don’t remember doing so. It seems as if they slowly lose the capacity to learn or update beliefs even slightly from this repetitive activity, and as this happens their sentience goes away. So we haven’t established that the sentient ball of uniform ecstasy is actually possible.
Meanwhile, a badly programmed AI might decide that a non-sentient or briefly-sentient ball still fits its programmed definition of the goal. Or it might think this about a ball that is just barely sentient.
One reason I might not want to become a ball of ecstasy is that I don’t really feel it’s ME. I’m not even sure it’s sentient, since it doesn’t learn, communicate, or reason, it just enjoys.
Sentience is exactly just the ability to feel. If it can feel joy, it is sentient.
Yes, but for example in highway hypnosis people drive on ‘boring’ stretches of highway and then don’t remember doing so. It seems as if they slowly lose the capacity to learn or update beliefs even slightly from this repetitive activity, and as this happens their sentience goes away. So we haven’t established that the sentient ball of uniform ecstasy is actually possible.
Meanwhile, a badly programmed AI might decide that a non-sentient or briefly-sentient ball still fits its programmed definition of the goal. Or it might think this about a ball that is just barely sentient.