It seems to me that Tell culture is unstable: if there is no social cost for stating strong preferences, people will be incentivized to overstate their preferences, and it will de facto reduce to Ask culture. If stating strong preferences is costly, people will be incentivized to undrestate their preferences, and you’ll end up with Guess culture.
Worse, everyone’s going to have a different set point for stated preference strength even if they don’t intentionally overstate or understate their preferences. We can get away with using verbiage like “rate X on a 1 to 5 scale” for collecting statistics, because set point variations will average out over a large enough data set—but that doesn’t fly for interpersonal interactions.
It seems to me that Tell culture is unstable: if there is no social cost for stating strong preferences, people will be incentivized to overstate their preferences, and it will de facto reduce to Ask culture.
This is false. This reduction only occurs if you also assume that the agents are also compelled to do some degree of sloppy utilitarian aggregation.
Stating strong preferences doesn’t compel people to give you more stuff. It tells people that to whatever extent they wish to give you stuff, which kind of stuff you would prefer to have.
Consider Bob:
Bob values being given a chocolate cake 9.
Bob values a foot massage 2.
Bob values being given a beer 1.
Bob could lie and say:
I value being given a chocolate cake OVER 9000!.
I value a foot massage OVER 9000!.
I value being given a beer OVER 9000!.
Consider Alice:
Alice giving a chocolate cake costs 4.
Alice giving a foot massage to Bob costs 3.
Alice giving a beer costs 2.
Alice wishes to spend 5 on Bob (either in trade or out of altruism) and to do so in the most effective way.
Since Bob has overstated Alice concludes that giving a foot massage and a beer will give Bob OVER 18,000! value while giving Bob a chocolate cake will give Bob a mere OVER 9,000! value.
If Bob had stated his preferences accurately then Alice would have given him the cake and he would have got his 9 value. He lied, so he got a mere 3 value. That was silly of him.
It isn’t Tell Culture that makes Bob’s communication devolve to Ask culture. Bob being bad at decision theory makes him devolve to Ask Culture. Alice should go hang out with Jack, who is less of an ineffectively-demanding-jackass.
But people do make a sort-of utilitarian calculation about what to give to whom. If your friend Xerxes says he values chocolate cake at OVER 9000!!! and your friend Ygnacio says he values it at 1, you care about both of them roughly equally, and you assume that they’re stating their preferences honestly, you should give the cake to Xerxes. But then there’s an incentive to exaggerate one’s preferences.
But people do make a sort-of utilitarian calculation about what to give to whom.
No, they really don’t. Most significantly because most people don’t behave as consequentialists of any kind. Also, and this time fortunately, our egalitarian instincts are sufficient to suppress that kind of folly.
If your friend Xerxes says he values chocolate cake at OVER 9000!!! and your friend Ygnacio says he values it at 1, you care about both of them roughly equally, and you assume that they’re stating their preferences honestly, you should give the cake to Xerxes.
No, you shouldn’t. Even if we ignore the type error of comparing Xerxes_value and Ygnacio_value and your decision ‘should’ take into account other information including things like who you gave the strawberry tarts to ten minutes ago and assorted other social transactions. I have not met a single human who gives all his favours to the same person because they are the most enthusiastic (and would consider the resultant behaviour to be repugnant and a behavioural red flag).
But then there’s an incentive to exaggerate one’s preferences.
Yes, as the grandparent observed, when the people being spoken to are both gullible and utilitarian and the speakers are neither ethical nor utilitarian ‘Tell’ does devolve into ‘Ask’. However if the listener is either not-gullible or not a crude total utilitarian then exaggerating your preferences amounts to crippling your own ability to receive value via either trade or gifts towards the limit of being only able to communicate booleans. (ie. It’s purely destructive self-sabotage.)
No, they really don’t. Most significantly because most people don’t behave as consequentialists of any kind.
Most people don’t consistently behave as consequentialists, but they do make consequentialist decisions some of the time, particularly in cases like this one.
Consider a less extreme example. Suppose your friend Xerxes is obsessed with Beethoven—he listens to every known composition and tries to learn it, and derives great enjoyment from doing so. Your friend Ygnacio also likes classical music in general but has no specific fondness for Beethoven. While digging in your belongings, you discover a sheet of antique sheet music personally written by Beethoven. Coincidentally, Xerxes’s and Ygnacio’s birthdays are coming up, and this would make a good gift for either of them—but as there’s only one sheet of music, only one of them can receive it. Certainly, Ygnacio would appreciate it, but Xerxes would like it much more. In such a situation, most people would give the sheet music to Xerxes, because he would enjoy it more.
As for the utility monster, that’s a nonsequitur in this context, because we’re not talking about true (agent-neutral) utilitarianism, only about utility maximization, which is not the same thing.
Even if we ignore the type error of comparing XerxesValue and YgnacioValue
We’re not comparing XerxesValue and YgnacioValue, we’re comparing HowMuchYouCareAboutXerxes x XerxesValue and HowMuchYouCareAboutYgnacio x YgnacioValue, which does not produce a type error.
your decision ‘should’ take into account other information including things like who you gave the strawberry tarts to ten minutes ago and assorted other social transactions
If you gave the strawberry tarts to someone ten minutes ago, it is reasonable to assume that because of diminishing marginal utility, they won’t value sweets as highly as they did before. But if you have reason to believe that they don’t experience diminishing marginal utility, or that their diminished derived utility would still be greater than the utility derived by an alternative person, then you should give it to the person who would derive greater utility (assuming you value them equally).
It’s true that people don’t always give all favors to the most enthusiastic person, but that is justified because it’s reasonable to assume that enthusiasm isn’t always a reliable indication of derived value.
(Had to edit this a million times, markup hates me.)
But if you have reason to believe that they don’t experience diminishing marginal utility, or that their diminished derived utility would still be greater than the utility derived by an alternative person, then you should give it to the person who would derive greater utility (assuming you value them equally).
How do you think caring about having more allies than one affects this situation?
Alice wishes to spend 5 on Bob (either in trade or out of altruism) and to do so in the most effective way.
This assumes that the amount of resource that Alice wants to spend on Bob is fixed. This is generally not the case. As blacktrance noted, usually there is competition. Even when there is no competition, Alice may be induced to spend more on Bob if he hides or misrepresents his true preferences: If Bob overstated, Alice might think she is making him a great favour, and hence she is building lots of social capital, when in fact she isn’t.
There are also scenarios where it is useful to understate one’s preferences: in a typical bargaining, for instance, the buyer often has an incentive to understate their preference for the good being traded (or overstate their preference for money), while the seller has an incentive to do the opposite.
In general, agents often have an incentive to reveal their preferences as little as possible, in order to exploit the information asymmetry.
In general, agents often have an incentive to reveal their preferences as little as possible, in order to exploit the information asymmetry.
I think that’s only the case in competitive games, not cooperative ones. (ISTM the optimal amount of information to reveal would be zero in the zero-sum-game limit and everything you know (neglecting the cost of communication itself etc.) in the identical-payoff-matrices limit.)
I think that’s only the case in competitive games, not cooperative ones.
A problem I continue to run into in real life is, “how do you keep people from wire-heading their preferences whenever they find themselves in a positive-sum game, so that they can play a zero-sum version instead?”
Original preference—“I just want a car to get to work.”
Environmental change: “Here, everyone gets a car for free.”
Adjusted preference—“Okay, then what I REALLY want is a faster car than anyone else has.”
...
Original preference—“I just want to be able to eat.”
Environmental change: “Here, there’s enough food to go around forever.”
Adjusted preference—“Okay, then what I REALLY want is for me to eat while those guys have to watch, starving.”
...
Original preference—“I just want to feel safe.”
Environmental change: “Here, you’re in a space where everyone is your ally and no one can hurt you.”
Adjusted preference—“Okay, then what I REALLY want is for us to all gang up on the people who made us feel unsafe, and make THEM feel persecuted for a change.”
Original preference—“I just want to be able to eat.”
Environmental change: “Here, there’s enough food to go around forever.”
Adjusted preference—“Okay, then what I REALLY want is for me to eat while those guys have to watch, starving.”
You mean “What I REALLY want is to eat better than everybody else”, surely. Gourmet food or organic or hand-prepared etc. etc.
Unless this is intended to imply the current famines etc. are the product of a conspiracy and not civilizational inadequacy, in which case yes, that would of course be evidence that my model of human nature is wrong an yours is the correct one, if it’s true.
The way you formulate this opposes the idea that the “adjusted” preference was actually the preference all along, and the originally stated preference was simply an incorrect description of the system’s actual preferences. Is that deliberate, or just an incidental artifact of your phrasing?
It’s an artifact of my phrasing. In my experience, people do truly want good things, until those things become universally available—at which point they switch goals to something zero-sum. When they do so, they often phrase it themselves as if they really wanted the zero-sum thing all along, but that’s often a part of trying to distance themselves from their lower-status past.
Of course, I’m describing something that I only have personal and anecdotal evidence for; I’d REALLY like to be pointed towards either a legitimate, peer-reviewed description of a cognitive bias that would explain what I’m observing. (And I’d be at least equally happy if it turned out to be my cognitive bias that’s causing me to perceive people in this way.)
In my experience, people do truly want good things, until those things become universally available—at which point they switch goals to something zero-sum.
What would you expect to experience differently if, instead, people truly want zero-sum things, but they claim to want good things until the universal availability of good things makes that claim untenable?
I’ll need some time to think on this. This might just be my tendency to find the most charitable interpretation, even if other interpretations might be more parsimonious.
When they do so, they often phrase it themselves as if they really wanted the zero-sum thing all along, but that’s often a part of trying to distance themselves from their lower-status past.
So it was deliberate, and not an artifact of your phrasing. Did you perhaps misread the grandparent?
(ISTM the optimal amount of information to reveal would be zero in the zero-sum-game limit and everything you know (neglecting the cost of communication itself etc.) in the identical-payoff-matrices limit.)
Interestingly, ISTM that is itself a Prisoner’s Dilemma: the agent that doesn’t reveal it’s (true) preferences has a much, much better chance of manipulating an agent that does.
Because you tend to always ask overstating your preferences, and the other party understands that you are probably overstating, hence preference claims lose informative value, and at some point they can just be dropped.
You seem to be claiming that in an Ask culture, if I say “I want X” I expect others to understand that I don’t actually want X, but rather want some other thing Y for which X is an overstatement, where Y doesn’t get stated explicitly.
In Tell culture you say something equivalent to “I want X with strength 9 out of 10”. The problem is that if everybody always says “9 out of 10″ the stated preference becomes meaningless and the message becomes “I want X”.
If stating strong preferences is costly, people will be incentivized to undrestate their preferences, and you’ll end up with Guess culture.
Tell culture with a preference weight cost certainly does devolve. It doesn’t devolve into guess culture (as described in the post), but into something different and arguably (but not strictly) worse. Guess culture works passably well when the asker/guesser is able to reliably model the response of the listener. This ‘stating preferences is costly’ system doesn’t handle that situation correctly.
Well, then… have a moderate social cost for stating strong preferences? I mean, it seems like you’re saying that it’s hard to get that balanced close enough that people can approximate optimal strategy as ‘just tell the truth’.
And you haven’t at all argued that the change in this cost over time would be away from balance rather than towards, which is what unstable means.
First off, you said it was unstable, which means there would have to be positive feedback to make it UN-stable.
Secondly, it seems to me that there is a source of negative feedback in that if people want to be able to express stronger preferences themselves, they might tolerate stronger preferences expressed from others.
Edited to Add Clarification:
We are attempting to calibrate the social cost of expressing strong preferences. The strength of preferences expressed will vary in normal encounters due to actual variance of preferences, variance in peoples’ habits of how to express those preferences, and variance in details of delivery.
If you would like to be freer to express your preferences more strongly, you can dial back the social costs you personally impose on others for expressing strong preferences. If they shift up their expressed preferences in response, then either you may as well or they are being hypocritical.
Conversely, if you think others are expressing preferences rather too strongly, first back off the strength of your own expressed preferences towards the desired level and then begin imposing more social costs on strong expressions of preferences.
Either do this gradually so people hardly notice, or out in the open, negotiated. Both options seem fairly intuitive and natural to me, though of course either one will itself impose social costs. Social skills help a lot in mitigating these costs.
First off, you said it was unstable, which means there would have to be positive feedback to make it UN-stable.
You don’t need a positive feedback to have instability.
Secondly, it seems to me that there is a source of negative feedback in that if people want to be able to express stronger preferences themselves, they might tolerate stronger preferences expressed from others.
Which means that there is an incentive to overstate preferences.
You don’t need a positive feedback to have instability.
Yes you do. Like, rigid pendulum at the top of its swing, F = +kx. That’s positive feedback. I suppose you can get around this requirement with discrete timesteps or other hackery, but classically speaking positive feedback <-> instability.
Which means that there is an incentive to overstate preferences.
… differentially so, from a starting point of understated preferences, so that’s a correcting change.
Okay, so that’s the definition of ‘unstable’ you were using. You’ve now taken care of the nitpick and left the main thrust of the argument unaddressed.
It seems to me that Tell culture is unstable: if there is no social cost for stating strong preferences, people will be incentivized to overstate their preferences, and it will de facto reduce to Ask culture. If stating strong preferences is costly, people will be incentivized to undrestate their preferences, and you’ll end up with Guess culture.
Worse, everyone’s going to have a different set point for stated preference strength even if they don’t intentionally overstate or understate their preferences. We can get away with using verbiage like “rate X on a 1 to 5 scale” for collecting statistics, because set point variations will average out over a large enough data set—but that doesn’t fly for interpersonal interactions.
For example, xkcd
This is false. This reduction only occurs if you also assume that the agents are also compelled to do some degree of sloppy utilitarian aggregation.
Stating strong preferences doesn’t compel people to give you more stuff. It tells people that to whatever extent they wish to give you stuff, which kind of stuff you would prefer to have.
Consider Bob:
Bob values being given a chocolate cake 9.
Bob values a foot massage 2.
Bob values being given a beer 1.
Bob could lie and say:
I value being given a chocolate cake OVER 9000!.
I value a foot massage OVER 9000!.
I value being given a beer OVER 9000!.
Consider Alice:
Alice giving a chocolate cake costs 4.
Alice giving a foot massage to Bob costs 3.
Alice giving a beer costs 2.
Alice wishes to spend 5 on Bob (either in trade or out of altruism) and to do so in the most effective way.
Since Bob has overstated Alice concludes that giving a foot massage and a beer will give Bob OVER 18,000! value while giving Bob a chocolate cake will give Bob a mere OVER 9,000! value.
If Bob had stated his preferences accurately then Alice would have given him the cake and he would have got his 9 value. He lied, so he got a mere 3 value. That was silly of him.
It isn’t Tell Culture that makes Bob’s communication devolve to Ask culture. Bob being bad at decision theory makes him devolve to Ask Culture. Alice should go hang out with Jack, who is less of an ineffectively-demanding-jackass.
But people do make a sort-of utilitarian calculation about what to give to whom. If your friend Xerxes says he values chocolate cake at OVER 9000!!! and your friend Ygnacio says he values it at 1, you care about both of them roughly equally, and you assume that they’re stating their preferences honestly, you should give the cake to Xerxes. But then there’s an incentive to exaggerate one’s preferences.
No, they really don’t. Most significantly because most people don’t behave as consequentialists of any kind. Also, and this time fortunately, our egalitarian instincts are sufficient to suppress that kind of folly.
No, you shouldn’t. Even if we ignore the type error of comparing Xerxes_value and Ygnacio_value and your decision ‘should’ take into account other information including things like who you gave the strawberry tarts to ten minutes ago and assorted other social transactions. I have not met a single human who gives all his favours to the same person because they are the most enthusiastic (and would consider the resultant behaviour to be repugnant and a behavioural red flag).
Yes, as the grandparent observed, when the people being spoken to are both gullible and utilitarian and the speakers are neither ethical nor utilitarian ‘Tell’ does devolve into ‘Ask’. However if the listener is either not-gullible or not a crude total utilitarian then exaggerating your preferences amounts to crippling your own ability to receive value via either trade or gifts towards the limit of being only able to communicate booleans. (ie. It’s purely destructive self-sabotage.)
Most people don’t consistently behave as consequentialists, but they do make consequentialist decisions some of the time, particularly in cases like this one. Consider a less extreme example. Suppose your friend Xerxes is obsessed with Beethoven—he listens to every known composition and tries to learn it, and derives great enjoyment from doing so. Your friend Ygnacio also likes classical music in general but has no specific fondness for Beethoven. While digging in your belongings, you discover a sheet of antique sheet music personally written by Beethoven. Coincidentally, Xerxes’s and Ygnacio’s birthdays are coming up, and this would make a good gift for either of them—but as there’s only one sheet of music, only one of them can receive it. Certainly, Ygnacio would appreciate it, but Xerxes would like it much more. In such a situation, most people would give the sheet music to Xerxes, because he would enjoy it more. As for the utility monster, that’s a nonsequitur in this context, because we’re not talking about true (agent-neutral) utilitarianism, only about utility maximization, which is not the same thing.
We’re not comparing XerxesValue and YgnacioValue, we’re comparing HowMuchYouCareAboutXerxes x XerxesValue and HowMuchYouCareAboutYgnacio x YgnacioValue, which does not produce a type error.
If you gave the strawberry tarts to someone ten minutes ago, it is reasonable to assume that because of diminishing marginal utility, they won’t value sweets as highly as they did before. But if you have reason to believe that they don’t experience diminishing marginal utility, or that their diminished derived utility would still be greater than the utility derived by an alternative person, then you should give it to the person who would derive greater utility (assuming you value them equally). It’s true that people don’t always give all favors to the most enthusiastic person, but that is justified because it’s reasonable to assume that enthusiasm isn’t always a reliable indication of derived value.
(Had to edit this a million times, markup hates me.)
How do you think caring about having more allies than one affects this situation?
If that’s a term in your utility function, then you should consider it. Here, I’m assuming there aren’t any other effects.
This assumes that the amount of resource that Alice wants to spend on Bob is fixed. This is generally not the case.
As blacktrance noted, usually there is competition.
Even when there is no competition, Alice may be induced to spend more on Bob if he hides or misrepresents his true preferences: If Bob overstated, Alice might think she is making him a great favour, and hence she is building lots of social capital, when in fact she isn’t.
There are also scenarios where it is useful to understate one’s preferences: in a typical bargaining, for instance, the buyer often has an incentive to understate their preference for the good being traded (or overstate their preference for money), while the seller has an incentive to do the opposite.
In general, agents often have an incentive to reveal their preferences as little as possible, in order to exploit the information asymmetry.
I think that’s only the case in competitive games, not cooperative ones. (ISTM the optimal amount of information to reveal would be zero in the zero-sum-game limit and everything you know (neglecting the cost of communication itself etc.) in the identical-payoff-matrices limit.)
A problem I continue to run into in real life is, “how do you keep people from wire-heading their preferences whenever they find themselves in a positive-sum game, so that they can play a zero-sum version instead?”
What does “wire-head a preference” mean?
Rewrite your utility function.
Examples:
Original preference—“I just want a car to get to work.”
Environmental change: “Here, everyone gets a car for free.”
Adjusted preference—“Okay, then what I REALLY want is a faster car than anyone else has.”
...
Original preference—“I just want to be able to eat.”
Environmental change: “Here, there’s enough food to go around forever.”
Adjusted preference—“Okay, then what I REALLY want is for me to eat while those guys have to watch, starving.”
...
Original preference—“I just want to feel safe.”
Environmental change: “Here, you’re in a space where everyone is your ally and no one can hurt you.”
Adjusted preference—“Okay, then what I REALLY want is for us to all gang up on the people who made us feel unsafe, and make THEM feel persecuted for a change.”
You mean “What I REALLY want is to eat better than everybody else”, surely. Gourmet food or organic or hand-prepared etc. etc.
Unless this is intended to imply the current famines etc. are the product of a conspiracy and not civilizational inadequacy, in which case yes, that would of course be evidence that my model of human nature is wrong an yours is the correct one, if it’s true.
The way you formulate this opposes the idea that the “adjusted” preference was actually the preference all along, and the originally stated preference was simply an incorrect description of the system’s actual preferences. Is that deliberate, or just an incidental artifact of your phrasing?
It’s an artifact of my phrasing. In my experience, people do truly want good things, until those things become universally available—at which point they switch goals to something zero-sum. When they do so, they often phrase it themselves as if they really wanted the zero-sum thing all along, but that’s often a part of trying to distance themselves from their lower-status past.
Of course, I’m describing something that I only have personal and anecdotal evidence for; I’d REALLY like to be pointed towards either a legitimate, peer-reviewed description of a cognitive bias that would explain what I’m observing. (And I’d be at least equally happy if it turned out to be my cognitive bias that’s causing me to perceive people in this way.)
What would you expect to experience differently if, instead, people truly want zero-sum things, but they claim to want good things until the universal availability of good things makes that claim untenable?
I’ll need some time to think on this. This might just be my tendency to find the most charitable interpretation, even if other interpretations might be more parsimonious.
What do you mean by “trully want”? See the phenomenon Eliezer describes here.
I intend the phrase to refer to whatever ialdabaoth meant by it when I quoted them.
So it was deliberate, and not an artifact of your phrasing. Did you perhaps misread the grandparent?
Interestingly, ISTM that is itself a Prisoner’s Dilemma: the agent that doesn’t reveal it’s (true) preferences has a much, much better chance of manipulating an agent that does.
If you know that the game is zero-sum then you usually already know all the other player preferences.
Can you clarify how overstating preferences reduces to Ask culture?
Because you tend to always ask overstating your preferences, and the other party understands that you are probably overstating, hence preference claims lose informative value, and at some point they can just be dropped.
You seem to be claiming that in an Ask culture, if I say “I want X” I expect others to understand that I don’t actually want X, but rather want some other thing Y for which X is an overstatement, where Y doesn’t get stated explicitly.
Have I understood you correctly?
No.
In Tell culture you say something equivalent to “I want X with strength 9 out of 10”.
The problem is that if everybody always says “9 out of 10″ the stated preference becomes meaningless and the message becomes “I want X”.
Ah! I now understand what you’re saying. Sure, that makes sense. Thanks for clarifying.
Tell culture with a preference weight cost certainly does devolve. It doesn’t devolve into guess culture (as described in the post), but into something different and arguably (but not strictly) worse. Guess culture works passably well when the asker/guesser is able to reliably model the response of the listener. This ‘stating preferences is costly’ system doesn’t handle that situation correctly.
Well, then… have a moderate social cost for stating strong preferences? I mean, it seems like you’re saying that it’s hard to get that balanced close enough that people can approximate optimal strategy as ‘just tell the truth’.
And you haven’t at all argued that the change in this cost over time would be away from balance rather than towards, which is what unstable means.
there doesn’t seem to be any obvious negative feedback to keep the balance stable.
First off, you said it was unstable, which means there would have to be positive feedback to make it UN-stable.
Secondly, it seems to me that there is a source of negative feedback in that if people want to be able to express stronger preferences themselves, they might tolerate stronger preferences expressed from others.
Edited to Add Clarification:
We are attempting to calibrate the social cost of expressing strong preferences. The strength of preferences expressed will vary in normal encounters due to actual variance of preferences, variance in peoples’ habits of how to express those preferences, and variance in details of delivery.
If you would like to be freer to express your preferences more strongly, you can dial back the social costs you personally impose on others for expressing strong preferences. If they shift up their expressed preferences in response, then either you may as well or they are being hypocritical.
Conversely, if you think others are expressing preferences rather too strongly, first back off the strength of your own expressed preferences towards the desired level and then begin imposing more social costs on strong expressions of preferences.
Either do this gradually so people hardly notice, or out in the open, negotiated. Both options seem fairly intuitive and natural to me, though of course either one will itself impose social costs. Social skills help a lot in mitigating these costs.
You don’t need a positive feedback to have instability.
Which means that there is an incentive to overstate preferences.
Yes you do. Like, rigid pendulum at the top of its swing, F = +kx. That’s positive feedback. I suppose you can get around this requirement with discrete timesteps or other hackery, but classically speaking positive feedback <-> instability.
… differentially so, from a starting point of understated preferences, so that’s a correcting change.
an unbiased random walk sufficies.
Okay, so that’s the definition of ‘unstable’ you were using. You’ve now taken care of the nitpick and left the main thrust of the argument unaddressed.
(edited for spelling)
Can you rephrase the the main thrust of the argument?
All right. To keep it from ending up at the leaf of this back-and-forth, I’ll edit-to-add it earlier on.