I don’t have good data to back this up, but I have a feeling that people are thinking in more binary terms than you expect. More specifically, I conjecture that if you were to ask someone for how likely meeting a certain politically charged event would be, they would parse your question as a binary one and answer either “almost certainly” or “very unlikely”—and when pressed for a number, would give you either between 90-100%, or 0-10% respectively.
Since you address “how likely meeting a certain politically charged event would be”, I assume your question is focussed on what I’ve called “Polling 2″, which concerns itself with predicting future events. These tend to be less politically charged than than “Polling 1”, but I agree you are right in pointing out the need to relativise respondents answers. People who identify strongly with a cause, especially if they are not used to dealing with probability, might confuse a question about an event likelihood with the strength of their allegiance. Thus “How likely do you think the Dodgers are to win the World Series?” might be met with “I’d bet my life on it”, which is not very helpful for computing statistics :-)
The best way to put the matter into quantitative terms may be to ask the interviewee what odds he would give in a bet on the event occuring. It may seem redundant, but I would also ask the odds they’d give on a non-occurence. (People’s grasp on probability is shaky, so overdeterminining their perception helps to reduce error).
You will notice that for Polling 1 type questions I avoided the natural step of asking people to say how much money it would take to get them to change their mind. For one thing, it would be tasteless to appear to be offering money to get someone to change a vote (for instance). Another reason is that people’s perceptions of money vary widely, injecting a confounding variable. The rather convoluted question I came up with to assess an interviewee’s resistance to chance of intent has the disadvantage of generating a discreet (non-continuous) answer. and I worry it might also confuse some interviewees, but at least it makes a quantification in terms of a comparable quantity.
Since you address “how likely meeting a certain politically charged event would be”, I assume your question is focussed on what I’ve called “Polling 2″, which concerns itself with predicting future events.
Yes, you’re right, and I should have been more clear—thanks for pointing that out.
The best way to put the matter into quantitative terms may be to ask the interviewee what odds he would give in a bet on the event occuring
I don’t know if I’m convinced that would work. I think that most people fall into two camps regarding betting odds. Camp A is not familiar with probability theory/calculus and doesn’t think in probabilistic terms in their daily life—they are only familiar with bets and odds as a rhetorical device (“the odds were stacked against him”, “against all odds”, etc). Camp B are people who bet/gamble as a pastime frequently, and are actually familiar with betting odds as an operable concept. If you ask an A about their odds on an event related to a cause they feel strongly about, they will default to their understanding of odds as a rhetorical device and signal allegiance instead of giving you a usable estimate. If you ask a B about their odds on the same event, they will start thinking about it in monetary terms, since habitual gamblers usually bet and gamble money. But, as you point out, putting a price on faith/belief/allegiance is seen as an immoral/dishonorable act, and would cause even more incentive to allegiance-signal instead of truly estimating probabilities. In this way, this only works either for surveying people with good skills at rationality/probability, or for surveying people about events they don’t have strong feelings on.
However, there are two pitfalls to this argument, and that’s why I’m not stating this with complete certainty. First, this is still speculation—I have no solid data on how familiar an aggregate (I’m not sure average is a good term to use here, given that mathematicians probably understand this concept very well while being relatively scarce) person is with concept of betting odds—and actually, do tell if you know any survey data that would allow to verify this, a gauge of how familiar people are with a certain concept seems like data useful enough to exist. Second, this may be cultural. I’m not American and have never stayed in America long-term—and based on you mentioning baseball and the time of your reply I assume that you are—so potentially the concept of betting is somehow more ingrained into the culture there, and I’m just defaulting to my prior knowledge.
The rather convoluted question I came up with to assess an interviewee’s resistance to chance of intent has the disadvantage of generating a discreet (non-continuous) answer. and I worry it might also confuse some interviewees
Yup. I didn’t see the point in highlighting that, since you mention yourself that the measure is imperfect, but this echoes my concern on betting. At the risk of sounding like an intellectualist snob, I think even the probabilistic concepts that most lesswrongers would see as basic are somewhat hard to imagine and operate with, save for as rhetorical devices, to the general public.
I don’t have good data to back this up, but I have a feeling that people are thinking in more binary terms than you expect. More specifically, I conjecture that if you were to ask someone for how likely meeting a certain politically charged event would be, they would parse your question as a binary one and answer either “almost certainly” or “very unlikely”—and when pressed for a number, would give you either between 90-100%, or 0-10% respectively.
Thanks, betulaster,
Since you address “how likely meeting a certain politically charged event would be”, I assume your question is focussed on what I’ve called “Polling 2″, which concerns itself with predicting future events. These tend to be less politically charged than than “Polling 1”, but I agree you are right in pointing out the need to relativise respondents answers. People who identify strongly with a cause, especially if they are not used to dealing with probability, might confuse a question about an event likelihood with the strength of their allegiance. Thus “How likely do you think the Dodgers are to win the World Series?” might be met with “I’d bet my life on it”, which is not very helpful for computing statistics :-)
The best way to put the matter into quantitative terms may be to ask the interviewee what odds he would give in a bet on the event occuring. It may seem redundant, but I would also ask the odds they’d give on a non-occurence. (People’s grasp on probability is shaky, so overdeterminining their perception helps to reduce error).
You will notice that for Polling 1 type questions I avoided the natural step of asking people to say how much money it would take to get them to change their mind. For one thing, it would be tasteless to appear to be offering money to get someone to change a vote (for instance). Another reason is that people’s perceptions of money vary widely, injecting a confounding variable. The rather convoluted question I came up with to assess an interviewee’s resistance to chance of intent has the disadvantage of generating a discreet (non-continuous) answer. and I worry it might also confuse some interviewees, but at least it makes a quantification in terms of a comparable quantity.
Yes, you’re right, and I should have been more clear—thanks for pointing that out.
I don’t know if I’m convinced that would work. I think that most people fall into two camps regarding betting odds. Camp A is not familiar with probability theory/calculus and doesn’t think in probabilistic terms in their daily life—they are only familiar with bets and odds as a rhetorical device (“the odds were stacked against him”, “against all odds”, etc). Camp B are people who bet/gamble as a pastime frequently, and are actually familiar with betting odds as an operable concept.
If you ask an A about their odds on an event related to a cause they feel strongly about, they will default to their understanding of odds as a rhetorical device and signal allegiance instead of giving you a usable estimate.
If you ask a B about their odds on the same event, they will start thinking about it in monetary terms, since habitual gamblers usually bet and gamble money. But, as you point out, putting a price on faith/belief/allegiance is seen as an immoral/dishonorable act, and would cause even more incentive to allegiance-signal instead of truly estimating probabilities.
In this way, this only works either for surveying people with good skills at rationality/probability, or for surveying people about events they don’t have strong feelings on.
However, there are two pitfalls to this argument, and that’s why I’m not stating this with complete certainty.
First, this is still speculation—I have no solid data on how familiar an aggregate (I’m not sure average is a good term to use here, given that mathematicians probably understand this concept very well while being relatively scarce) person is with concept of betting odds—and actually, do tell if you know any survey data that would allow to verify this, a gauge of how familiar people are with a certain concept seems like data useful enough to exist.
Second, this may be cultural. I’m not American and have never stayed in America long-term—and based on you mentioning baseball and the time of your reply I assume that you are—so potentially the concept of betting is somehow more ingrained into the culture there, and I’m just defaulting to my prior knowledge.
Yup. I didn’t see the point in highlighting that, since you mention yourself that the measure is imperfect, but this echoes my concern on betting. At the risk of sounding like an intellectualist snob, I think even the probabilistic concepts that most lesswrongers would see as basic are somewhat hard to imagine and operate with, save for as rhetorical devices, to the general public.