People generally care more about furthering personal pleasure and minimizing personal pain than the pleasure/pain of others; but this is because internal personal pleasure was a straightforward good heuristic for evolution to take when it wanted to maximize genetic fitness in the ancestral environment where there weren’t that many sudden out-of-distribution things (like contraceptives) that could derail it.
I assume a more strongly-optimized intelligent being would have increasingly better correlation between the state of its internal utility to the state of the external world, as it fits whatever goal it was optimized for better. In that case it should more readily collaborate with its clone.
This especially if it gets optimized with other instances of itself so that “cloning” is no longer a weird out-of-distribution event; in which case I expect it to rapidly start behaving like an ant or bee, or even cell or mitochondria, in how it’ll sacrifice itself for whatever goal the group has.
These are some very good points. We are so used to human-human interactions that it is easy to assume that this is some kind of universal way in which agents interact.
People generally care more about furthering personal pleasure and minimizing personal pain than the pleasure/pain of others; but this is because internal personal pleasure was a straightforward good heuristic for evolution to take when it wanted to maximize genetic fitness in the ancestral environment where there weren’t that many sudden out-of-distribution things (like contraceptives) that could derail it.
I assume a more strongly-optimized intelligent being would have increasingly better correlation between the state of its internal utility to the state of the external world, as it fits whatever goal it was optimized for better. In that case it should more readily collaborate with its clone.
This especially if it gets optimized with other instances of itself so that “cloning” is no longer a weird out-of-distribution event; in which case I expect it to rapidly start behaving like an ant or bee, or even cell or mitochondria, in how it’ll sacrifice itself for whatever goal the group has.
These are some very good points. We are so used to human-human interactions that it is easy to assume that this is some kind of universal way in which agents interact.