I think you are pointing at something real, but the slices you make here don’t quite carve reality at the joints as I see it. I don’t have a better alternative cohesive hypothesis to offer as an alternative, just a feeling like this needs further discussion and refinement before I fully agree.
Here’s a piece of what I’ve been thinking about which I do think fits pretty well into this model.
Important concept for what I want to express:
BATNA—Best Alternative To Negotiated Agreement
Sometimes negotiations are implicit rather than explicit. Like growing up as a child in a particular society, and having a particular social contract thrust upon you.
What is the meaning of negotiation when non-human actors are part of it?
Does the lion negotiate with an antelope when the antelope jumps performatively to indicate its ability to outrun the lion and the lion chooses an easier prey? This is more like game theory than negotiation.
A rebellion can be seen as a BATNA to continuing to submit to the rule of a government.
What if you do negotiate with a scheming AI and successfully negotiate a peaceful arrangement which is safe for both parties? How does this forecast humanity’s ability to accomplish similar negotiations with future stronger AI?
I’m beating around the bush here because I think it’s important to widen people’s thought patterns on this subject. I strongly agree with you that talking about negotiating ownership of fractions of lightcone or galaxy are absurd.
Please consider the following thought experiment: a group of relatively uneducated humans with IQs ranging between 90 and 110 gets cursed by a witch and magically transformed into a grove of oak trees. These trees have the same “minds” but operate on a slower time frame. Each is capable of thinking at a rate of about 1 word per quarter year, and communicating at a rate of 1 word per year.
Now suppose that before their transformation these people were part of a functional democracy which represented them reasonably well, in line with modern democracies. They had lots of complaints, but they were citizens with rights and votes.
How well do you think the democracy of which they were a minority fraction continues to represent them going forward?
When people talk about negotiating with future AIs, I also imagine large intelligence differences, but I think that that is harder for people to imagine and plan around. So instead, focus on just this speed difference thought experiment.
Imagine you are a hunted slave, running for freedom. You realize you are unlikely to escape. You have a communication device through which you are able to speak to your pursuers. You negotiate with them to surrender in return for some guarantees of Rights, and manage to loop in some third party arbiters in some way such that you feel you can trust the bargain. In the distance you see your pursuers all turn into oak trees. Do you hold to the terms of your negotiated capture and lifelong enslavement to them? I think not. The BATNA just changed dramatically.
I think you are pointing at something real, but the slices you make here don’t quite carve reality at the joints as I see it. I don’t have a better alternative cohesive hypothesis to offer as an alternative, just a feeling like this needs further discussion and refinement before I fully agree.
Here’s a piece of what I’ve been thinking about which I do think fits pretty well into this model.
Important concept for what I want to express: BATNA—Best Alternative To Negotiated Agreement
Sometimes negotiations are implicit rather than explicit. Like growing up as a child in a particular society, and having a particular social contract thrust upon you.
What is the meaning of negotiation when non-human actors are part of it? Does the lion negotiate with an antelope when the antelope jumps performatively to indicate its ability to outrun the lion and the lion chooses an easier prey? This is more like game theory than negotiation.
A rebellion can be seen as a BATNA to continuing to submit to the rule of a government.
What if you do negotiate with a scheming AI and successfully negotiate a peaceful arrangement which is safe for both parties? How does this forecast humanity’s ability to accomplish similar negotiations with future stronger AI?
I’m beating around the bush here because I think it’s important to widen people’s thought patterns on this subject. I strongly agree with you that talking about negotiating ownership of fractions of lightcone or galaxy are absurd.
Please consider the following thought experiment: a group of relatively uneducated humans with IQs ranging between 90 and 110 gets cursed by a witch and magically transformed into a grove of oak trees. These trees have the same “minds” but operate on a slower time frame. Each is capable of thinking at a rate of about 1 word per quarter year, and communicating at a rate of 1 word per year.
Now suppose that before their transformation these people were part of a functional democracy which represented them reasonably well, in line with modern democracies. They had lots of complaints, but they were citizens with rights and votes.
How well do you think the democracy of which they were a minority fraction continues to represent them going forward?
When people talk about negotiating with future AIs, I also imagine large intelligence differences, but I think that that is harder for people to imagine and plan around. So instead, focus on just this speed difference thought experiment.
Imagine you are a hunted slave, running for freedom. You realize you are unlikely to escape. You have a communication device through which you are able to speak to your pursuers. You negotiate with them to surrender in return for some guarantees of Rights, and manage to loop in some third party arbiters in some way such that you feel you can trust the bargain. In the distance you see your pursuers all turn into oak trees. Do you hold to the terms of your negotiated capture and lifelong enslavement to them? I think not. The BATNA just changed dramatically.