I don’t have a link, but I think the basic idea is that sufficiently smart entities should be able to calculate the probabilities and costs of victory, and negotiate a better outcome.
Something like: if you and me are fighting for a bag of coins, and we both agree that my probability to win the bag is 60% and your probability is 40%, it would make more sense to stop fighting and split the bad 60:40. But perhaps you have a hidden knife I don’t know about, because you planned to use it only if I am clearly winning the fight. If we both are smart enough, we might negotiate something like you saying “there is something you don’t know about, can you credibly promise that if I disclose that information to you, you will take it into account when calculating the probabilities, rather than abusing the fact that now you know the information” and then we might split it 20:80 or something.
In summary: knowing the probabilities and costs, and having a way to precommit to handling previously unknown information in a fair way (to agree on the estimates of probabilities and costs had the information remained unknown).
Does anyone know Yudkowsky’s elaboration on ASIs not doing combat against each other?
I don’t have a link, but I think the basic idea is that sufficiently smart entities should be able to calculate the probabilities and costs of victory, and negotiate a better outcome.
Something like: if you and me are fighting for a bag of coins, and we both agree that my probability to win the bag is 60% and your probability is 40%, it would make more sense to stop fighting and split the bad 60:40. But perhaps you have a hidden knife I don’t know about, because you planned to use it only if I am clearly winning the fight. If we both are smart enough, we might negotiate something like you saying “there is something you don’t know about, can you credibly promise that if I disclose that information to you, you will take it into account when calculating the probabilities, rather than abusing the fact that now you know the information” and then we might split it 20:80 or something.
In summary: knowing the probabilities and costs, and having a way to precommit to handling previously unknown information in a fair way (to agree on the estimates of probabilities and costs had the information remained unknown).