Yes, it’s possible to strike deals, but that doesn’t mean they will actually be ‘friendly’, at most ‘neutral’. They may superficially give off the appearance of being ‘friendly’ but then again humans do that too all the time.
By this token, everyone is neutral, no one is friendly, unless I am literally their top priority in the whole world, and they mine (that doesn’t sound like simple “friendship”… more like some kind of eternal fated bond between soulmates). For me friendly is someone with whom there’s ample and reasonable chances to communicate and establish a common ground. If you make conditions harsh enough friends can turn into rivals for survival, but that’s why we want abundance and well-being to be the norm. However, no amount of abundance will satisfy a relentless maximizer—they will always want more, and never stop. That’s what makes compromise with them impossible. Humans are more like satisficers.
By this token, everyone is neutral, no one is friendly, unless I am literally their top priority in the whole world, and they mine (that doesn’t sound like simple “friendship”… more like some kind of eternal fated bond between soulmates).
Can you explain your reasoning here? How does a bias towards or against imply a ‘top priority’?
Well, you’re the one who’s saying “the aliens wouldn’t be friendly if they know you’re biased towards your own side”. A bias means I prioritize my race over the aliens. This is normal and pretty expected; the aliens, too, will surely prioritize their own race over humans, if push came to shove. That’s no barrier to friendship. The ability to cooperate is fundamentally dependent on circumstances. The only case in which I will be absolutely sure that someone would never, ever turn on me, no matter how dire the circumstances, is if I am their top priority. Bias means you have a hierarchy of values, and some are higher than others; so “well-being of your family” is higher than “well-being of an equivalent number of total strangers”, and “well-being of humanity” may be higher than “well-being of the sentient octopuses of Rigel-4”. But the world usually isn’t made of binary trolley problems, and agents that are willing to be friendly and to put each other at a reasonably high (but not necessarily top) position in their value hierarchies have plenty of occasions to establish fruitful collaboration by throwing some other, less important value under the bus.
A relentless maximizer however is a fundamentally selfish kind of agent. A maximizer can never truly compromise because it does not have a range of acceptable states—it has only ONE all-important value that defines a single acceptable target state, and all its actions are in service of achieving that state. It can perform friendship only as long as it serves its goal, and will backstab you the next moment even if it was not in existential danger, merely because it has to advance towards its goal. I may care for the well-being of my wife, but I am not a Wife-Well-Being Maximizer. I would not for example stab someone to steal a pair of earrings that she would like if only I could get away with it; I still value a stranger’s life far more than my wife’s marginal enjoyment from a new piece of jewellery. A maximizer instead only cares about the goal, and everything else is at best instrumental, which makes it fundamentally unreliable (unless YOUR well-being happens to be the goal it cares about maximizing, and even then, I’d consider it a risky agent to have around).
Yes, it’s possible to strike deals, but that doesn’t mean they will actually be ‘friendly’, at most ‘neutral’. They may superficially give off the appearance of being ‘friendly’ but then again humans do that too all the time.
By this token, everyone is neutral, no one is friendly, unless I am literally their top priority in the whole world, and they mine (that doesn’t sound like simple “friendship”… more like some kind of eternal fated bond between soulmates). For me friendly is someone with whom there’s ample and reasonable chances to communicate and establish a common ground. If you make conditions harsh enough friends can turn into rivals for survival, but that’s why we want abundance and well-being to be the norm. However, no amount of abundance will satisfy a relentless maximizer—they will always want more, and never stop. That’s what makes compromise with them impossible. Humans are more like satisficers.
Can you explain your reasoning here? How does a bias towards or against imply a ‘top priority’?
Well, you’re the one who’s saying “the aliens wouldn’t be friendly if they know you’re biased towards your own side”. A bias means I prioritize my race over the aliens. This is normal and pretty expected; the aliens, too, will surely prioritize their own race over humans, if push came to shove. That’s no barrier to friendship. The ability to cooperate is fundamentally dependent on circumstances. The only case in which I will be absolutely sure that someone would never, ever turn on me, no matter how dire the circumstances, is if I am their top priority. Bias means you have a hierarchy of values, and some are higher than others; so “well-being of your family” is higher than “well-being of an equivalent number of total strangers”, and “well-being of humanity” may be higher than “well-being of the sentient octopuses of Rigel-4”. But the world usually isn’t made of binary trolley problems, and agents that are willing to be friendly and to put each other at a reasonably high (but not necessarily top) position in their value hierarchies have plenty of occasions to establish fruitful collaboration by throwing some other, less important value under the bus.
A relentless maximizer however is a fundamentally selfish kind of agent. A maximizer can never truly compromise because it does not have a range of acceptable states—it has only ONE all-important value that defines a single acceptable target state, and all its actions are in service of achieving that state. It can perform friendship only as long as it serves its goal, and will backstab you the next moment even if it was not in existential danger, merely because it has to advance towards its goal. I may care for the well-being of my wife, but I am not a Wife-Well-Being Maximizer. I would not for example stab someone to steal a pair of earrings that she would like if only I could get away with it; I still value a stranger’s life far more than my wife’s marginal enjoyment from a new piece of jewellery. A maximizer instead only cares about the goal, and everything else is at best instrumental, which makes it fundamentally unreliable (unless YOUR well-being happens to be the goal it cares about maximizing, and even then, I’d consider it a risky agent to have around).