Then, if one agent sees the other agent performing poorly, then why would the first agent decide not to help the second one with the task (e.g. by giving hints which prevent the mental process from circulating in the wrong region of ideas) instead of stealing resources?
Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.
If stealing isn’t commonly used or thought about, then what moral situation can reveal the difference between stealing being unused and stealing being objectively bad? Is it stealing from another civilisation?
Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.
If stealing isn’t commonly used or thought about, then what moral situation can reveal the difference between stealing being unused and stealing being objectively bad? Is it stealing from another civilisation?