Does the following make sense: A possible intuition pump would be to imagine a specific superintelligent mind with no consciousness, some kind of maximally stripped-down superintelligent algorithm, and ask whether the contents of its objective function constitute a “morality”. The answer to this question is more obviously “no” than if you vaguely wonder about “possible minds”.
Where “superintelligent” doesn’t mean “anthropomorphic”, but it literally means “a spider that is astronomically more intelligent at being a spider”. Like, building webs that use quantum physics. Using chaos theory to predict the movement of flies. Mad spider science beyond our imagination.
But when it comes to humans, the superintelligent spider cares about them exactly as much as a normal spider, that is not at all, unless they become a threat. Of course the superintelligent spider is better able to imagine how humans could become a threat in the future, and is better at using its spider science to eliminate humans. But it’s still a spider with spider values (not some kind of a human mind trapped in a fluffy eight-legged body). It might overcome the limitations of the spider body, and change its form into a machine or whatever. But its mind would remain (an extrapolation of) a spider mind.
Does the following make sense: A possible intuition pump would be to imagine a specific superintelligent mind with no consciousness, some kind of maximally stripped-down superintelligent algorithm, and ask whether the contents of its objective function constitute a “morality”. The answer to this question is more obviously “no” than if you vaguely wonder about “possible minds”.
Or maybe imagine a superintelligent spider.
Where “superintelligent” doesn’t mean “anthropomorphic”, but it literally means “a spider that is astronomically more intelligent at being a spider”. Like, building webs that use quantum physics. Using chaos theory to predict the movement of flies. Mad spider science beyond our imagination.
But when it comes to humans, the superintelligent spider cares about them exactly as much as a normal spider, that is not at all, unless they become a threat. Of course the superintelligent spider is better able to imagine how humans could become a threat in the future, and is better at using its spider science to eliminate humans. But it’s still a spider with spider values (not some kind of a human mind trapped in a fluffy eight-legged body). It might overcome the limitations of the spider body, and change its form into a machine or whatever. But its mind would remain (an extrapolation of) a spider mind.