Even more sinister, maybe: suppose it said there’s a level of processing on which you automatically interpret things in an intentional frame (ala Dan Dennet) and this ability to “intentionalize” things effectively simulates suffering/minds all the time in everyday objects in your environment, and that further, while we can correct it in our minds, this anthropomorphic projection happens as a necessary product, somehow, of our consciousness. Consciousness as we know it IS suffering and to create an FAI that won’t halt the moment it figures out that it is causing harm with its own thought processes, we’ll need to think really, really far outside the box.
Even more sinister, maybe: suppose it said there’s a level of processing on which you automatically interpret things in an intentional frame (ala Dan Dennet) and this ability to “intentionalize” things effectively simulates suffering/minds all the time in everyday objects in your environment, and that further, while we can correct it in our minds, this anthropomorphic projection happens as a necessary product, somehow, of our consciousness. Consciousness as we know it IS suffering and to create an FAI that won’t halt the moment it figures out that it is causing harm with its own thought processes, we’ll need to think really, really far outside the box.