It suggests to me that human values don’t depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.
By human values we mean how we treat things that are not part of the competitive environment.
The greatness of a nation and its moral progress can be judged by the way its animals are treated.
-- Mahatma Gandhi
Obviously a paperclip maximizer wouldn’t punch you in the face if you could destroy it. But if it is stronger than all other agents and doesn’t expect to ever having to prove its benevolence towards lesser agents, then there’ll be no reason to care about them? The only reason I could imagine for a psychopathic agent to care about agents that are less powerful is if there is some benefit in being friendly towards them. For example if there are a lot of superhuman agents out there and general friendliness enables cooperation and makes you less likely to be perceived as a threat and subsequently allows you to use less resources to fight.
By human values we mean how we treat things that are not part of the competitive environment.
-- Mahatma Gandhi
Obviously a paperclip maximizer wouldn’t punch you in the face if you could destroy it. But if it is stronger than all other agents and doesn’t expect to ever having to prove its benevolence towards lesser agents, then there’ll be no reason to care about them? The only reason I could imagine for a psychopathic agent to care about agents that are less powerful is if there is some benefit in being friendly towards them. For example if there are a lot of superhuman agents out there and general friendliness enables cooperation and makes you less likely to be perceived as a threat and subsequently allows you to use less resources to fight.
I don’t think I mean that. I also don’t know where you’re going with this observation.
Roughly, that you can specify human values by supplying a diff from optimal selfish competition.