I notice that I am confused. Suppose that two agents are trying to achieve different parts of a goal. Then, if one agent sees the other agent performing poorly, then why would the first agent decide not to help the second one with the task (e.g. by giving hints which prevent the mental process from circulating in the wrong region of ideas) instead of stealing resources? Additionally, the murder and theft from coding agents from your example are cheap, not good, because such an agent is describable by the LLM’s weights (which don’t actually disappear! So what does it even mean to murder or steal from an LLM agent?), its CoT (edit: and external documents) and potential cache values. Were creation of a coding agent to be genuinely hard, like requiring a human brain to practice coding for years, then there would be no reason to murder the agent, there would be a reason to give the failer a new task.
P.S. Stealing at larger scales is likely reframable as being good for the whole collective. For example, if, in a counterfactual world, Anthropic had its lobbyists across the USG (edit: and was genuinely better than GDM and xAI), then it might have been a good idea to destroy xAI, confiscate its compute and sell it to Anthropic.
Then, if one agent sees the other agent performing poorly, then why would the first agent decide not to help the second one with the task (e.g. by giving hints which prevent the mental process from circulating in the wrong region of ideas) instead of stealing resources?
Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.
If stealing isn’t commonly used or thought about, then what moral situation can reveal the difference between stealing being unused and stealing being objectively bad? Is it stealing from another civilisation?
I notice that I am confused. Suppose that two agents are trying to achieve different parts of a goal. Then, if one agent sees the other agent performing poorly, then why would the first agent decide not to help the second one with the task (e.g. by giving hints which prevent the mental process from circulating in the wrong region of ideas) instead of stealing resources? Additionally, the murder and theft from coding agents from your example are cheap, not good, because such an agent is describable by the LLM’s weights (which don’t actually disappear! So what does it even mean to murder or steal from an LLM agent?), its CoT (edit: and external documents) and potential cache values. Were creation of a coding agent to be genuinely hard, like requiring a human brain to practice coding for years, then there would be no reason to murder the agent, there would be a reason to give the failer a new task.
P.S. Stealing at larger scales is likely reframable as being good for the whole collective. For example, if, in a counterfactual world, Anthropic had its lobbyists across the USG (edit: and was genuinely better than GDM and xAI), then it might have been a good idea to destroy xAI, confiscate its compute and sell it to Anthropic.
Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.
If stealing isn’t commonly used or thought about, then what moral situation can reveal the difference between stealing being unused and stealing being objectively bad? Is it stealing from another civilisation?