Multiple Choice
When we choose behavior, including verbal behavior, it’s sometimes tempting to do what is most likely to be right without paying attention to how costly it is to be wrong in various ways or looking for a safer alternative.
If you’ve taken a lot of standardized tests, you know that some of them penalize guessing and some don’t. That is, leaving a question blank might be better than getting a wrong answer, or they might have the same result. If they’re the same, of course you guess, because it can’t hurt and may help. If they take off points for wrong answers, then there’s some optimal threshold at which a well-calibrated test-taker will answer. For instance, the ability to rule out one of four choices on a one-point question where a wrong answer costs a quarter point means that you should guess from the remaining three—the expected point value of this guess is positive. If you can rule out one of four choices and a wrong answer costs half a point, leave it blank.
If you have ever asked a woman who wasn’t pregnant when the baby was due, you might have noticed that life penalizes guessing.
If you’re risk-neutral, you still can’t just do whatever has the highest chance of being right; you must also consider the cost of being wrong. You will probably win a bet that says a fair six-sided die will come up on a number greater than 2. But you shouldn’t buy this bet for a dollar if the payoff is only $1.10, even though that purchase can be summarized as “you will probably gain ten cents”. That bet is better than a similarly-priced, similarly-paid bet on the opposite outcome; but it’s not good.
There’s a few factors at work to make guessing tempting anyway:
You can’t always leave “questions” blank.
Guessing is inconsistently penalized, and the penalties are often hidden.
Inaction sometimes has negative consequences too.
An increased ability to rule things out relative to the general population causes overconfidence.
We are biased against rating the value of information highly, and prefer to act on what we already know.
However, I maintain that we should refrain from guessing at a higher rate than baseline. This is especially relevant when choosing verbal behavior—we may remember to act according to expected utility, but rarely think to speak according to expected utility, and this is especially significant around sensitive topics where being careless with words can cause harm.
Taking on each reason to guess one at a time:
You can’t always leave questions blank, and unlike on a a written test, the “blank” condition is not always obvious. The fact that sometimes there is no sane null action—it’s typically not a good idea to stare vacantly at someone when they ask you a question, for instance—doesn’t mean, however, that there is never a sane null action. You can be pretty well immune to Dutch books simply by refusing to bet—this might cost gains when you don’t have Dutch-bookable beliefs, but it will prevent loss. It is worthwhile to train yourself to notice when it is possible to simply do nothing, especially in cases where you have a history of performing worse-than-null actions. For instance, I react with panic when someone explains something to me and I miss a step in their inference. I find I get better results if, instead of interrupting them and demanding clarification, I wait five seconds. This period of time is often enough for the conversation to provide the context for me to understand on my own, and if it doesn’t, at least it’s not enough of a lag that either of us will have forgotten what was said. But remembering that I can wait five seconds is hard.
Guessing is inconsistently penalized, with sometimes hidden costs, so it’s hard to extinguish the guessing response. If you’re prone to doing something in a certain situation, and doing that something doesn’t immediately sock you in the face every single time, it will take far longer for you to learn not to do it—this goes for people as well as pets. Both the immediate response and the subjective consistency are important, and a hidden cost contributes to neither. However, smart people can rise to the challenge of reacting to inconsistent, non-immediate, concealed costs to their actions. For instance, I’d be willing to bet that Less Wrong readers smoke less than the general population. Observe the relative frequency with which guessing hurts or may hurt you, compared to not guessing, and make your plans based on that.
Inaction can be harmful too, and there’s a psychological sting to something bad happening because you stood there and did nothing, especially when you’re familiar with the hazards of the bystander effect. I do not advocate spending all day sitting on the sofa for fear of acting wrongly while your house collapses around your ears. My point is that there are many situations where we guess and shouldn’t, not that we should never guess, and that there is low-hanging fruit in reducing bad guessing.
You’re right more often than a regular person, but that doesn’t mean you are right enough: life’s not graded on a curve. The question isn’t, Do I have a better shot at getting this one right than the neighbor across the street? but, Do I have a good enough shot at getting this one right? You can press your relative advantages when your opponents are people, but not when you’re playing against the house. (Additionally, you might have specific areas of comparative disadvantage even if you’re overall better than the folks down the road.)
Information is more valuable than it seems, and there is often a chance to try to improve one’s odds rather than settling for what one starts with. Laziness and impatience are both factors here: checking all the sources you could takes a long time and is harder than guessing. But in high stakes situations, it’s at least worth Googling, probably worth asking a couple people who might have expertise, maybe worth looking a bit harder for data on similar scenarios and recommended strategies for them. Temporal urgency is more rarely the factor in discounting the value of information-gathering than is simply wanting the problem to be over with; but problems are not really over with if they are guessed at wrongly, only if you get them right.
Maybe I’m misunderstanding what kinds of cases you have in mind, but I would’ve guessed that the bias is in the opposite direction. People tend to play it safe, choosing inaction, avoiding the worst possible outcome, and not sticking their neck out in the way that could leave them most visibly and unambiguously wrong and open to criticism.
Could you give some specific examples of cases where people seem to be biased towards guessing?
I agree with this. For example, there was a discussion a while back on opt-in versus opt-out organ donation. Most LWers supported opt-out because the downside of making a mistake in opt-out is a dead person’s preferences being accidentally violated, as opposed to the downside of making a mistake in opt-in, where someone dies because they don’t get an organ.
Most people in the general population prefer opt-in, and the only reason I can think of is that they feel like bad consequences through inaction (not taking an organ that should be taken) is okay in a way that bad consequences through action (taking an organ that shouldn’t be taken) is not.
Or, regarding literal multiple choice tests where the bias is the other way, see Shut Up And Guess.
Conversations with foreigners?
If you can rule out one of four choices, you’ll get the right answer 1⁄3 of the time, and a wrong answer 2⁄3 of the time, for an expected value of 1(1/3) - (1/2)(2/3) = 0, so it doesn’t matter whether you guess or not.
It was stated that a wrong answer costs a quarter point, so the expected value is 1(1/3) - (1/4)(2/3) = 1⁄6. A cost of 1⁄4 point for a wrong answer is too low though, since in that case the expected value of guessing from all four choices is still 1⁄16. The cost should be 1⁄3 to make that expectation equal 0.
Filling in the bubble costs time.
Inversely, filling in the bubble prevents you from getting one cell off for the following questions.
If you can firmly rule out one of the 4 options, it seems pretty unlikely that you are literally indifferent between the other three, so guessing is almost certainly positive expectation.
I was thinking about this very topic the other day in the shower. I had a bar of soap that I wanted to lob across the bathroom into a small box I usually keep it in. Between the box and me was, of course, an open toilet bowl.
I felt that my natural inclination was to overshoot—to error on the side of safety. I realized this and thought that I should instead suppress my fear of failure to maximize my chance of success. I then realized that that is a popular meme in our culture, especially as it relates to athletic feats. Popular memes make me suspicious, so a second’s more thought convinced me that the correct path is to maximize expected utility, not chance of success, and that fishing my soap out of a bachelor’s toilet presented a rather negative value in that calculation. I concluded overshooting was best and noted that the natural, fear induced response was thus a well honed heuristic. I lamented, simultaneously, that the popular meme was probably far stronger in many people (particularly athletes) than the instinctual response, and that the necessary rationality skills to overcome this were probably lacking. I attempted to generalize my result to rock climbing, where the negative outcome is potentially death, but stopped when I realized I was about to conclude that my own rationality skills must be lacking.
Satisfied with my analysis, I then attempted the shot with the full intent of making it, the toilet be damned.
Don’t leave us hanging!
I’m having some difficulty parsing what should count as “not guessing” in various iRL situations. There’s a certain degree to which I do not find that, or null actions in general, a natural category.
I do kinda get the concept in some situations. Like if you don’t know anything about the facts of some particular political controversy, then it’s probably for the best not to declare your support for either side before you’ve given the matter more thought. But it’s still a guess. You’re guessing that not declaring support for either side will mean that you’re more likely to not end up making statements you’ll regret later.
I wonder if this difference has something to do with looking at “actions” at differing levels. Like Alicorn looking more at the action itself, and me looking more at the thought process generating that action, or something along those lines. (It seems intuitively that whatever it is, it’s either the same thing or caused by the same thing that’s making her more of a deontologist and me more of a consequentialist, but I can’t quite put my finger on it.)
(EDIT: rewrote crappy comment)
This article really doesn’t seem to present anything that we don’t already know. The main thrust seems to be something like “consider expected utility when taking actions”, mixed in with some I don’t know and the null action is an action. However, I believe that putting this general knowledge into a specific context adds value, in that it helps the reader examine parts of rationality in an applied context.
But it seems like it could have more links to related concepts. They’re surely in there.
Allow me to be bewildered by the downvotes out loud, and request elaboration.
Edit: What I was curious about was why the downvoters disliked thomblake’s comment, although I wouldn’t mind knowing their objections to mine.
Well, why I would’ve downvoted that comment, is that it was basically just praise wrapped in word salad. I had a substantive point to make, but I failed to make it.
Is there a particular practical upshot, or an example that you were thinking of? I think the principle of “maximize expected utility, rather than assume the most probable outcome” is pretty mainstream already around here.
The “including verbal behavior” bit in the summary was really what I had in mind. I think that while we may act according to expected utility, we rarely remember to speak according to expected utility, and this is especially significant around sensitive topics where being careless with words can cause harm.
Trouble is, people tend to behave like real-life Nozick’s utility monsters when it comes to the disutility they profess to suffer from speech that offends their sensibilities. I am not a utilitarian, but even if I were, I don’t see how I could ever bring myself to speak (or rather keep quiet) according to expected utility.
(This is a statement about people in general, not meant as a jab at any concrete persons here.)
Utilitarians should take into account all consequences, including the cost of creating incentives to become utility monsters.
I think there’s probably an imbalance between the amount of effort people are expected to expend in not being offensive vs the amount of effort people are expected to expend in not being offended. I’m not sure where it comes from.
steven0461:
The practical impossibility of taking into account such game-theoretical considerations and other important indirect effects of decisions is one of the (less important) reasons why I see little to no worth in utilitarianism.
Nevertheless, your point is very good.
I don’t think you can work towards not being offended. {according to my very narrow definition, which I now retract} It’s just a gut reaction.
You can choose whether to nurse your offense or not nurse it, and you can choose whether to suggest to others that they should be offended. Reactions that are involuntary in the moment itself are sometimes voluntary in the longer run.
Taking offense is a tactic in politics and social interaction however as in ‘the politics of offense’. People will tend to use the tactic more when it appears to be successful.
Get a group of friends where you constantly make (facetious) offensive remarks at one another’s expense, both about individual qualities and group identifications. Eventually, having been called a filthy whatever-ethnicity or a loathsome whatever-sexual-orientation (including loathsome heterosexual, hey why not?) or a Christ-killer or a baby-eating atheist so many times, the emotional impact of such statements will be dulled, which will improve your ability to understand your actual objections and react usefully when you hear people seriously say such things. Worked for me!
I think it’s possible to work towards not being offended by such things as remembering the times when one was accidentally offensive, by checking on whether one’s standards are reasonable, and by evaluating actual risks of what the offensive thing might indicate.
That doesn’t mean one can or should run one’s offendedness down to zero, but (depending one where you’re starting), it’s possible to town it down.
Speech is hella hard to be rational about because the penalty for inaction when speaking in groups is high—if you value having a chance to speak, and if your group isn’t using a formal scheduling algorithm to give all would-be speakers their time slice. You can end up entirely excluded from the conversation just for hesitating. Therefore in most circumstances people speak “from cache”, so it seems to me.
Ever hung out with a group of friends who used a talking stick or similar token? If so, was the average quality of conversation much higher than with other groups of similar size and median IQ?
Great! I find that comment more concise, and more easy to apply, than the post.
I’ll edit it in, then.
Harm to ourselves or to others? Estimating expected utility is pretty complicated in cases where we have to weigh up upsetting others (according to the value we directly assign to others’ utility in our utility functions), upsetting others (according to the harm we estimate to ourselves through creating enemies) and benefiting ourselves (by rapidly screening off others who we don’t wish to interact with). I imagine typical individuals are generally doing all of these calculations at some level in typical interactions and it is not clear to me that we consistently err in any particular direction.
I think this probably varies a lot depending on the person.
Huh. Any examples on your mind? I can’t form a concrete idea of what’s to be avoided.
I actually wrote all but the last two paragraphs of this post weeks and weeks ago, so I can’t remember what individual things I had in mind at the time. I assign about a 65% chance that I was thinking about discussions of pickup or something similar.
That was helpful insight, thanks.
“You will probably gain ten cents” is not a summary of all the relevant information. If money and utility are interchangeable at this small scale (if you don’t desperately need an extra 10¢ but you’d prefer to increase the amount of money you have) then an expected utility calculation can tell you that it’s a bad bet — it comes out to an expected loss of 26.6¢.
edit: here’s an example:
You have a 1⁄3 chance of losing a dollar and a 2⁄3 chance of gaining ten cents. On average, you will lose 13 (edit for, um, the fifth time: 26) cents per dollar. Unless you need that dime to buy a ticket for the last plane leaving your doomed volcanic island home… it’s a bad bet.
Also see: applause lights