Yes, but for example in highway hypnosis people drive on ‘boring’ stretches of highway and then don’t remember doing so. It seems as if they slowly lose the capacity to learn or update beliefs even slightly from this repetitive activity, and as this happens their sentience goes away. So we haven’t established that the sentient ball of uniform ecstasy is actually possible.
Meanwhile, a badly programmed AI might decide that a non-sentient or briefly-sentient ball still fits its programmed definition of the goal. Or it might think this about a ball that is just barely sentient.
Yes, but for example in highway hypnosis people drive on ‘boring’ stretches of highway and then don’t remember doing so. It seems as if they slowly lose the capacity to learn or update beliefs even slightly from this repetitive activity, and as this happens their sentience goes away. So we haven’t established that the sentient ball of uniform ecstasy is actually possible.
Meanwhile, a badly programmed AI might decide that a non-sentient or briefly-sentient ball still fits its programmed definition of the goal. Or it might think this about a ball that is just barely sentient.