Note that there are two very distinct reasons for cooperation/negotation:
1) It’s the best way to get what I want. The better I model other agents, the better I can predict how to interact with them in a way that meets my desires. For this item, an external agent is no different from any other complex system.
2) I actually care about the other agent’s well-being. There is a term in my utility function for their satisfaction.
Very weirdly, we tend to assume #2 about humans (when it’s usually a mix of mostly 1 and a bit of 2). And we focus on #1 for AI, with no element of #2.
When you say “code for cooperation”, I can’t tell if you’re just talking about #1, or some mix of the two, where caring about the other’s satisfaction is a goal.
In humans, there’s a lot of #2 behind our cooperative ability (even if the result looks a lot like #1). I don’t know how universal that will be, but it seems likely to be computationally cheaper at some margin to encode #2 than to calculate and prove #1.
In my view, “code for cooperation” will very often have a base assumption that cooperation in satisfying others’ goals is more effective (which feels like “more pleasant” or “more natural” from inside the algorithm) than contractual resource exchanges.
Note that there are two very distinct reasons for cooperation/negotation:
1) It’s the best way to get what I want. The better I model other agents, the better I can predict how to interact with them in a way that meets my desires. For this item, an external agent is no different from any other complex system. 2) I actually care about the other agent’s well-being. There is a term in my utility function for their satisfaction.
Very weirdly, we tend to assume #2 about humans (when it’s usually a mix of mostly 1 and a bit of 2). And we focus on #1 for AI, with no element of #2.
When you say “code for cooperation”, I can’t tell if you’re just talking about #1, or some mix of the two, where caring about the other’s satisfaction is a goal.
Mostly #1. Is there a reason to build AIs that inherently care about the well-being of paperclippers etc?
But EA should be mostly #2?
In humans, there’s a lot of #2 behind our cooperative ability (even if the result looks a lot like #1). I don’t know how universal that will be, but it seems likely to be computationally cheaper at some margin to encode #2 than to calculate and prove #1.
In my view, “code for cooperation” will very often have a base assumption that cooperation in satisfying others’ goals is more effective (which feels like “more pleasant” or “more natural” from inside the algorithm) than contractual resource exchanges.