I’m sure this observation has been made plenty of times before: a principal can gain negotiating power by delegating negotiations to an agent, and restricting that agent’s ability to negotiate.
Well that sure can’t be an equilibrium of a completed timeless decision theory with reflective consistency. Your delegees are more powerful because they have fewer choices? Why wouldn’t you just rewrite your source code to eliminate those options? Why wouldn’t you just not do them? And why would the other agent react any differently to the delegate than to the source-code change or the decision in the moment?
In hindsight, whoever gave my comment its initial “-1 point” ding was correct: although I thought “Why wouldn’t you just rewrite your source code” was a flippant question that doesn’t mean it deserved just a joking answer. So, some more serious answers:
Your delegates are more powerful because they are known to have fewer choices and because they are known to value those choices differently, which can prevent them from being subject to threats or affected by precommitments that might have been useful against you.
I wouldn’t rewrite my source code because, as I joked, I can’t.. but even if I could, doing so would only be effective if there were some way of also convincing other agents that I wasn’t deceiving them about my new source code. This may not be practical: for every program that does X when tested, returns source code for “do X” when requested, and does X in the real world, there exists another program which does X when tested, returns source code for “do X” when requested, and does Y in the real world. See the concern over electronic voting machines for a more contemporary example of the problem.
Whether I would just not do something is irrelevant—what matters is whether everyone interacting with me believes I will do it. It’s easier for a customer to believe that a cashier won’t exceed his authority than for a customer to believe that an owner won’t accept a still-mutually-beneficial bargain, even if the owner swears that he precommitted not to haggle.
Wild speculation: There are instances where evolution seems to have built “one-boxing” type adaptations into humanity, and in those cases we seem to find precommitment claims plausible. If someone is hurt badly enough then they may want revenge even if taking revenge hurts them further. If someone is treated generously enough then they may be generous in return despite not wanting anything further from their benefactor. A lot of the “irrational” emotions look a lot like rational precommitments from the right perspective. But if you find yourself wishing you could precommit in a situation where apes aren’t known for precommitting, it might be too late—the precommitment only helps if it’s believed. Delegation is one of the ways you can make a precommitment more believable.
Someone really should write a “Cliffs Notes for Schelling” sequence. I’d naturally prefer “someone else”, but if nobody starts it by December I suppose I’ll try writing an intro post in January.
Why wouldn’t you just rewrite your source code to eliminate those options?
While corporations don’t have literal source code to modify, operating under a set of procedures that visibly make negotiation impossible, such as having the customer interact with an employee who is not authorized to negotiate, does essentially what you are saying.
Your delegees are more powerful because they have fewer choices? Why wouldn’t you just rewrite your source code to eliminate those options?
Well, you are more powerful because your delegees have fewer choices. “Delegate negotiations to an agent with different source code” seems equivalent to “rewrite your source code” (assuming the agent can’t communicate with you on demand.)
Actually, it seems possibly even more general, since you are always free to revoke the agent later.
As to why the agent would react differently: all other things being equal it wouldn’t. However, we do have the inbuilt instinct to go to irrational lengths against those who try to cheat us, and “corporation delegating to an agent” doesn’t feel like cheating because it’s standard. I suspect that “precommitment not to negotiate”, depending on how it’s expressed, would instinctively look much more like a kind of cheating to most people.
I’m sure this observation has been made plenty of times before: a principal can gain negotiating power by delegating negotiations to an agent, and restricting that agent’s ability to negotiate.
Well that sure can’t be an equilibrium of a completed timeless decision theory with reflective consistency.
You’re right which means that the answer to the question:
And why would the other agent react any differently to the delegate than to the source-code change or the decision in the moment?
… is “People are crazy; the world is mad.”
The mistake is to conclude that vulnerability to (or dependence on) this kind of tactic must be part of decision theory rather than just something that is effective for most humans.
And why would the other agent react any differently to the delegate than to the source-code change or the decision in the moment?
Let’s go a bit more meta...
The world is imperfect. And we all know it. Therefore, when faced with an imperfection that seems inevitable, we often forgive it.
But people don’t have correct models of the world, so they can’t distinguish reliably between evitable and inevitable imperfections. This can be exploited by creating imperfections which seem inevitable, and which “coincidentally” increase your negotiating power.
For example if you hire agents to represent you, your customers usually can’t tell the difference between the instructions you had to give them (e.g. because of the imperfections of the agents, or possible conflicts between you and the agents), and the instructions you have them deliberately to make life more difficult for your customers. Sometimes your customers even don’t know whether you really had to hire the agents, or you just chose to do so because it gave you a leverage.
The answer is in some form: Customers don’t have full knowledge about what really happened, which includes knowledge about how much their lack of knowledge was used against them.
Aren’t you just neglecting that humans can’t self-modify much?
No, and in particular certainly not just. Even if we decided that “read about some decision theory and better understand how to make decisions” doesn’t qualify as “change your source code” the other option of “just not do them” requires no change.
My point is that in some cases the “option of “just not do them”” does require a change (if you count precommitting devices and the like as changes). There are people who wouldn’t be able to successfully resolve to (say) just stop smoking, they’d have to somehow prevent their future selves from doing so—which does count as a change IMO.
My point is that in some cases the “option of “just not do them”” does require a change (if you count precommitting devices and the like as changes). There are people who wouldn’t be able to successfully resolve to (say) just stop smoking
I understand what your are saying about akrasia and maintain that the intended rhetorical point of your question is not especially relevant to it’s context. You are arguing against a position I wouldn’t support so increasingly detailed explanations of something that was trivial to begin with aren’t especially useful.
Obviously quitting smoking counts as change and involves enormous akrasia problems. An example of something that doesn’t count as changing is just not negotiating in a certain situation because you are one of the many people who are predisposed to just not negotiate in such situations. That actually means not changing instead of changing (in response to pressure from a naive decision theory or naive decision theorist that asserts that negotiating is the rational choice when precommitment isn’t possible.)
The problem with MixedNut’s claim:
Aren’t you just neglecting that humans can’t self-modify much?
… wasn’t that humans in fact can self modify a lot (they can’t). The problem was that this premise doesn’t weaken Eliezer’s point significantly even though it is true.
Well that sure can’t be an equilibrium of a completed timeless decision theory with reflective consistency. Your delegees are more powerful because they have fewer choices? Why wouldn’t you just rewrite your source code to eliminate those options? Why wouldn’t you just not do them? And why would the other agent react any differently to the delegate than to the source-code change or the decision in the moment?
Rewriting my source code is tricky; I always start to get dizzy from the blood loss before the saw is even halfway through my skull.
In hindsight, whoever gave my comment its initial “-1 point” ding was correct: although I thought “Why wouldn’t you just rewrite your source code” was a flippant question that doesn’t mean it deserved just a joking answer. So, some more serious answers:
Your delegates are more powerful because they are known to have fewer choices and because they are known to value those choices differently, which can prevent them from being subject to threats or affected by precommitments that might have been useful against you.
I wouldn’t rewrite my source code because, as I joked, I can’t.. but even if I could, doing so would only be effective if there were some way of also convincing other agents that I wasn’t deceiving them about my new source code. This may not be practical: for every program that does X when tested, returns source code for “do X” when requested, and does X in the real world, there exists another program which does X when tested, returns source code for “do X” when requested, and does Y in the real world. See the concern over electronic voting machines for a more contemporary example of the problem.
Whether I would just not do something is irrelevant—what matters is whether everyone interacting with me believes I will do it. It’s easier for a customer to believe that a cashier won’t exceed his authority than for a customer to believe that an owner won’t accept a still-mutually-beneficial bargain, even if the owner swears that he precommitted not to haggle.
Wild speculation: There are instances where evolution seems to have built “one-boxing” type adaptations into humanity, and in those cases we seem to find precommitment claims plausible. If someone is hurt badly enough then they may want revenge even if taking revenge hurts them further. If someone is treated generously enough then they may be generous in return despite not wanting anything further from their benefactor. A lot of the “irrational” emotions look a lot like rational precommitments from the right perspective. But if you find yourself wishing you could precommit in a situation where apes aren’t known for precommitting, it might be too late—the precommitment only helps if it’s believed. Delegation is one of the ways you can make a precommitment more believable.
Someone really should write a “Cliffs Notes for Schelling” sequence. I’d naturally prefer “someone else”, but if nobody starts it by December I suppose I’ll try writing an intro post in January.
While corporations don’t have literal source code to modify, operating under a set of procedures that visibly make negotiation impossible, such as having the customer interact with an employee who is not authorized to negotiate, does essentially what you are saying.
Well, you are more powerful because your delegees have fewer choices. “Delegate negotiations to an agent with different source code” seems equivalent to “rewrite your source code” (assuming the agent can’t communicate with you on demand.)
Actually, it seems possibly even more general, since you are always free to revoke the agent later.
As to why the agent would react differently: all other things being equal it wouldn’t. However, we do have the inbuilt instinct to go to irrational lengths against those who try to cheat us, and “corporation delegating to an agent” doesn’t feel like cheating because it’s standard. I suspect that “precommitment not to negotiate”, depending on how it’s expressed, would instinctively look much more like a kind of cheating to most people.
You’re right which means that the answer to the question:
… is “People are crazy; the world is mad.”
The mistake is to conclude that vulnerability to (or dependence on) this kind of tactic must be part of decision theory rather than just something that is effective for most humans.
Let’s go a bit more meta...
The world is imperfect. And we all know it. Therefore, when faced with an imperfection that seems inevitable, we often forgive it.
But people don’t have correct models of the world, so they can’t distinguish reliably between evitable and inevitable imperfections. This can be exploited by creating imperfections which seem inevitable, and which “coincidentally” increase your negotiating power.
For example if you hire agents to represent you, your customers usually can’t tell the difference between the instructions you had to give them (e.g. because of the imperfections of the agents, or possible conflicts between you and the agents), and the instructions you have them deliberately to make life more difficult for your customers. Sometimes your customers even don’t know whether you really had to hire the agents, or you just chose to do so because it gave you a leverage.
The answer is in some form: Customers don’t have full knowledge about what really happened, which includes knowledge about how much their lack of knowledge was used against them.
Aren’t you just neglecting that humans can’t self-modify much?
No, and in particular certainly not just. Even if we decided that “read about some decision theory and better understand how to make decisions” doesn’t qualify as “change your source code” the other option of “just not do them” requires no change.
Have you ever heard of akrasia?
Akrasia is one of thousands of things that I have heard of that do not seem particularly salient to the point.
I mean, among humans “just not doing things” takes, you know, willpower.
Yes, that is what akrasia means. I reaffirm both my ancestor comments.
My point is that in some cases the “option of “just not do them”” does require a change (if you count precommitting devices and the like as changes). There are people who wouldn’t be able to successfully resolve to (say) just stop smoking, they’d have to somehow prevent their future selves from doing so—which does count as a change IMO.
I understand what your are saying about akrasia and maintain that the intended rhetorical point of your question is not especially relevant to it’s context. You are arguing against a position I wouldn’t support so increasingly detailed explanations of something that was trivial to begin with aren’t especially useful.
Obviously quitting smoking counts as change and involves enormous akrasia problems. An example of something that doesn’t count as changing is just not negotiating in a certain situation because you are one of the many people who are predisposed to just not negotiate in such situations. That actually means not changing instead of changing (in response to pressure from a naive decision theory or naive decision theorist that asserts that negotiating is the rational choice when precommitment isn’t possible.)
The problem with MixedNut’s claim:
… wasn’t that humans in fact can self modify a lot (they can’t). The problem was that this premise doesn’t weaken Eliezer’s point significantly even though it is true.