I’ve established a habit of putting my money where my mouth is to encourage myself to make more firm predictions. When I am talking with someone and we disagree, I ask if they want to bet a dollar on it. For example, I say, “Roger Ebert directed Beyond the Valley of the Dolls”. My wife says, “No, he wrote it.”. Then I offer to bet. We look up the answer, and I give her a dollar.
This is a good habit for many reasons.
It is fun to bet. Fun to win. And (kinda) fun to lose.
It forces people to evaluate honestly. The same people that say “I’m sure...” will back off their point when asked to bet a dollar on the outcome.
It forces people to negotiate to concrete terms. For example, I told a friend that a 747 burns 30,000 lbs of fuel an hour. He said no way. We finally settled on the bet “Ben thinks that a fully loaded 747 will burn more than 10,000 lbs of fuel per hour under standard cruising conditions”. (I won that bet, it burns ~25,000 lbs of fuel/hour under these conditions).
A dollar feels more important than it actually is, so people treat the bets seriously even though they are not very serious. For this reason, I think it is important to actually exchange a dollar bill at the end, rather than treating it as just an abstract dollar.
I’ve learned a lot from this habit.
I’m right more often than not (~75%). But I’m wrong a lot too (~25%). This is more wrong than I feel. I feel 95% confident. I shouldn’t be so confident.
The person proposing the bet is usually right. My wife has gotten in the habit too. If I propose we bet, I’m usually right. If she proposes we bet I’ve learned to usually back down.
People frequently overstate their confidence. I mentioned this above, but it bears repeating. Some people regularly will use phrases like “I am sure” or say something emphatically. People are calibrated to express their beliefs differently. But when you ask them to bet a dollar you get a more consistent calibration. People that are over-confident often back away from their positions. Really interesting considering that its only a dollar on the line.
Over time people learn to calibrate better. At first my wife would agree to nearly every bet I proposed. Now she usually doesn’t want to. When she agrees to a bet now, I get worried.
This is a crucial observation if you are trying to use this technique to improve your calibration of your own accuracy! You can’t just start making bets when no one else you associate regularly is challenging you to the bets.
Several years ago, I started taking note of all of the times I disagreed with other people and looking it up, but initially, I only counted myself as having “disagreed with other people” if they said something I thought was wrong, and I attempted to correct them. Then I soon added in the case when they corrected me and I argued back. During this period of time, I went from thinking I was about 90% accurate in my claims to believing I was way more accurate than that. I would go months without being wrong, and this was in college, so I was frequently getting into disagreements with people, probably, an average, three a day during the school year. Then I started checking the times that other people corrected me, just as much as I checked when I corrected other people. (Counting even the times that I made no attempt to argue.) And my accuracy rate plummeted.
Another thing I would recommend to people starting out in doing this is that you should keep track of your record with individual people not just your general overall record. My accuracy rate with a few people is way lower than my overall accuracy rate. My overall rate is higher than it should be because I know a few argumentative people who are frequently wrong. (This would probably change if we were actually betting money, and we were only counting arguments when those people were willing to bet. So you’re approach adjusts for this better than mine.) I have several people for whom I’m close to 50%, and there are two people for whom I have several data points and my overall accuracy is below 50%.
There’s one other point I think somebody needs to make about calibration. And that’s that 75% accuracy when you disagree with other people is not the same thing as 75% accuracy. 75% information fidelity is atrocious; 95% information fidelity is not much better. Human brains are very defective in a lot of ways, but they aren’t that defective! Except at doing math. Brains are ridiculously bad at math relative to how easily machines can be implemented to be good at math. For most intents and purposes, 99% isn’t a very high percentage. I am not a particular good driver, but I haven’t gotten into a collision with another vehicle in my well over 1000 times driving. Percentages tend to have an exponential scale to them (or more accurately a logistic curve). You don’t have to be a particularly good driver to avoid getting into an accident 99.9% of the time you get behind the wheel, because that is just a few orders of magnitude improvement relative to 50%.
Information fidelity differs from information retention. Discarding 25% or 95% or more of collected information is reasonable; corrupting information at that rate is what I’m saying would be horrendous. (Because discarding information conserves resources; whereas corrupting information does not… except to the extent that you would consider compressing information with a lossy (as in “not lossless”) compression to be a corrupting information, but I would still consider that to be discarding information. Episodic memory is either very compressed or very corrupted depending on what you think it should be.)
In my experience, people are actually more likely to be underconfident about factual information than they are to be overconfident, if you measure confidence on an absolute scale instead of a relative-to-other-people scale. My family goes to trivia night, and we almost always get at least as many correct as we expect to get correct, usually more. However, other teams typically score better than we expect them to score too, and we win the round less often than we expect to.
Think back to grade school when you actually had fill in the blank and multiple choice questions on tests. I’m going to guess that you probably were an A student and got around 95% right on your tests… because a) that’s about what I did and I tend to project, b) you’re on LessWrong so you were probably an A student, and C) you say you feel like you ought to be right about 95% of the time. I’m also going to guess (because I tend to project my experience onto other people) that you probably felt a lot less than 95% confident on average when you were taking the tests. There were more than a few tests I took in my time in school where I walked out of the test thinking “I didn’t know any of that; I’ll probably get a 70 or better just because that would be horribly bad compared to what I usually do, but I really feel like I failed that”… and it was never 70. (Math was the one exception in which I tended to be overconfident, I usually made more mistakes than I expected to make on my math tests.)
Where calibration is really screwed up is when you deal with subjects that are way outside of the domain of normal experience, especially if you know that you know more than your peer group about this domain. People are not good at thinking about abstract mathematics, artificial intelligence, physics, evolution, and other subjects that happen at a different scale from normal everyday life. When I was 17, I thought I understood Quantum Mechanics just because I’d read A Brief History of Time and A Universe in a Nut Shell… Boy was I wrong!
On LessWrong, we are usually discussing subjects that are way beyond the domain of normal human experience, so we tend to be overconfident in our understanding of these subjects… but part of the reason for this overconfidence is that we do tend to be correct about most of the things we encounter within the confines of routine experience.
I was reading your comment, and when I thought about always betting a dollar, my brain went, “That’s a good idea!”
So I asked my brain, “What memory are you accessing that makes you think that’s a good idea?”
And my brain replied, “Remember that CFAR reading list you’re going through? Yeah, that one.”
So I went to my bookshelf, got out Dan Ariely’s Predictably Irrational, and started paging through it.
Professor Ariely had several insights that helped me understand why actually using money seemed like such a good idea:
Interacting within market norms makes you do a cost-benefit analysis. Professor Ariely discusses the difference between social norms and market norms in chapter 4. Social norms govern interactions that don’t involve money (favors for a friend), and market norms govern interactions that do (costs and benefits). The professor did an experiment in which he had people drag circles into a square on a computer screen (judging their productivity by how many times they did this in a set period of time). He gave one group of such participants an explicitly stated “50-cent snickers bar” and the other a “five-dollar box of Godiva chocolates.” As it turns out, the results were identical to a previous experiment in which the same amounts of direct cash were used. Professor Ariely concludes, “These results show that for market norms to emerge, it is sufficient to mention money.”
In other words, Professor Ariely’s research supports your first (4.) - “A dollar feels more important than it actually is...” This is the case because as soon as money enters the picture, so do market norms.
Money makes us honest. In chapter 14, aptly titled “Why Dealing With Cash Makes Us More Honest,” Professor Ariely explains an experiment he conducted in the MIT cafeteria. Students were given a sheet of 20 math problems to solve in five minutes. The control group was to have their solutions checked, and then were given 50 cents per correct answer. A second group was instructed to tear their paper apart, and then tell the experimenter how many questions they got correct (allowing them to cheat). They were then paid 50 cents for every correct answer they claimed. Lastly, a third group was allowed to cheat similarly to the second group, except that when they gave one experimenter their score, they were given tokens, which were traded in immediately thereafter for cash through a second experimenter.
The results:
A) The control group solved an average of 3.5 questions correctly.
B) The second group, who cheated for cash, claimed an average of 6.2 correct solutions.
C) The third group, who cheated for tokens, claimed an average of 9.4 correct solutions.
Simply put, when actual, physical money was removed from the subjects’ thought process by a token and a few seconds, the amount of cheating more than doubled, from 2.7 to 5.9.
In short, using money to back a prediction a) forces us to think analytically, and b) keeps us honest.
Thank you for the idea. Now I just need to find an ATM to get some ones...
This experiment does not prove that money keeps people more honest than absence of money, but more honest than token exchangeable for money. If a control group was allowed to cheat without receiving money at all they might (my prediction and I would bet a dollar on it if I didn’t use Euros) cheat even less. Then, the hypothesis “money keeps us honest” would be disproved.
I think I remember described set of experiments correctly and at least in some of them control group was definitely allowed to cheat—there were no difference in the way people turned in their results (shredding questionare and submitting only purported result on different sheet)
When I am talking with someone and we disagree, I ask if they want to bet a dollar on it. For example, I say, “Roger Ebert directed Beyond the Valley of the Dolls”. My wife says, “No, he wrote it.”. Then I offer to bet. We look up the answer, and I give her a dollar.
What would you do if he wrote and directed it? Neither? If there was more than one writer (which he was one of), or more than one director (of which he was one)?
A dollar feels more important than it actually is, so people treat the bets seriously even though they are not very serious.
Although there is a weight in the dollar, I think there is also another reason why people take it more seriously. People adjust their believe according to what other people believe and their confidence level. Therefore, when you propose a bet, even only for a dollar, you are showing a high confidence level and this decrease their confidence level. As a result, system 2 kicks in and they will be > [forced] to evaluate honestly.
I’ve established a habit of putting my money where my mouth is to encourage myself to make more firm predictions. When I am talking with someone and we disagree, I ask if they want to bet a dollar on it. For example, I say, “Roger Ebert directed Beyond the Valley of the Dolls”. My wife says, “No, he wrote it.”. Then I offer to bet. We look up the answer, and I give her a dollar.
This is a good habit for many reasons.
It is fun to bet. Fun to win. And (kinda) fun to lose.
It forces people to evaluate honestly. The same people that say “I’m sure...” will back off their point when asked to bet a dollar on the outcome.
It forces people to negotiate to concrete terms. For example, I told a friend that a 747 burns 30,000 lbs of fuel an hour. He said no way. We finally settled on the bet “Ben thinks that a fully loaded 747 will burn more than 10,000 lbs of fuel per hour under standard cruising conditions”. (I won that bet, it burns ~25,000 lbs of fuel/hour under these conditions).
A dollar feels more important than it actually is, so people treat the bets seriously even though they are not very serious. For this reason, I think it is important to actually exchange a dollar bill at the end, rather than treating it as just an abstract dollar.
I’ve learned a lot from this habit.
I’m right more often than not (~75%). But I’m wrong a lot too (~25%). This is more wrong than I feel. I feel 95% confident. I shouldn’t be so confident.
The person proposing the bet is usually right. My wife has gotten in the habit too. If I propose we bet, I’m usually right. If she proposes we bet I’ve learned to usually back down.
People frequently overstate their confidence. I mentioned this above, but it bears repeating. Some people regularly will use phrases like “I am sure” or say something emphatically. People are calibrated to express their beliefs differently. But when you ask them to bet a dollar you get a more consistent calibration. People that are over-confident often back away from their positions. Really interesting considering that its only a dollar on the line.
Over time people learn to calibrate better. At first my wife would agree to nearly every bet I proposed. Now she usually doesn’t want to. When she agrees to a bet now, I get worried.
This is a crucial observation if you are trying to use this technique to improve your calibration of your own accuracy! You can’t just start making bets when no one else you associate regularly is challenging you to the bets.
Several years ago, I started taking note of all of the times I disagreed with other people and looking it up, but initially, I only counted myself as having “disagreed with other people” if they said something I thought was wrong, and I attempted to correct them. Then I soon added in the case when they corrected me and I argued back. During this period of time, I went from thinking I was about 90% accurate in my claims to believing I was way more accurate than that. I would go months without being wrong, and this was in college, so I was frequently getting into disagreements with people, probably, an average, three a day during the school year. Then I started checking the times that other people corrected me, just as much as I checked when I corrected other people. (Counting even the times that I made no attempt to argue.) And my accuracy rate plummeted.
Another thing I would recommend to people starting out in doing this is that you should keep track of your record with individual people not just your general overall record. My accuracy rate with a few people is way lower than my overall accuracy rate. My overall rate is higher than it should be because I know a few argumentative people who are frequently wrong. (This would probably change if we were actually betting money, and we were only counting arguments when those people were willing to bet. So you’re approach adjusts for this better than mine.) I have several people for whom I’m close to 50%, and there are two people for whom I have several data points and my overall accuracy is below 50%.
There’s one other point I think somebody needs to make about calibration. And that’s that 75% accuracy when you disagree with other people is not the same thing as 75% accuracy. 75% information fidelity is atrocious; 95% information fidelity is not much better. Human brains are very defective in a lot of ways, but they aren’t that defective! Except at doing math. Brains are ridiculously bad at math relative to how easily machines can be implemented to be good at math. For most intents and purposes, 99% isn’t a very high percentage. I am not a particular good driver, but I haven’t gotten into a collision with another vehicle in my well over 1000 times driving. Percentages tend to have an exponential scale to them (or more accurately a logistic curve). You don’t have to be a particularly good driver to avoid getting into an accident 99.9% of the time you get behind the wheel, because that is just a few orders of magnitude improvement relative to 50%.
Information fidelity differs from information retention. Discarding 25% or 95% or more of collected information is reasonable; corrupting information at that rate is what I’m saying would be horrendous. (Because discarding information conserves resources; whereas corrupting information does not… except to the extent that you would consider compressing information with a lossy (as in “not lossless”) compression to be a corrupting information, but I would still consider that to be discarding information. Episodic memory is either very compressed or very corrupted depending on what you think it should be.)
In my experience, people are actually more likely to be underconfident about factual information than they are to be overconfident, if you measure confidence on an absolute scale instead of a relative-to-other-people scale. My family goes to trivia night, and we almost always get at least as many correct as we expect to get correct, usually more. However, other teams typically score better than we expect them to score too, and we win the round less often than we expect to.
Think back to grade school when you actually had fill in the blank and multiple choice questions on tests. I’m going to guess that you probably were an A student and got around 95% right on your tests… because a) that’s about what I did and I tend to project, b) you’re on LessWrong so you were probably an A student, and C) you say you feel like you ought to be right about 95% of the time. I’m also going to guess (because I tend to project my experience onto other people) that you probably felt a lot less than 95% confident on average when you were taking the tests. There were more than a few tests I took in my time in school where I walked out of the test thinking “I didn’t know any of that; I’ll probably get a 70 or better just because that would be horribly bad compared to what I usually do, but I really feel like I failed that”… and it was never 70. (Math was the one exception in which I tended to be overconfident, I usually made more mistakes than I expected to make on my math tests.)
Where calibration is really screwed up is when you deal with subjects that are way outside of the domain of normal experience, especially if you know that you know more than your peer group about this domain. People are not good at thinking about abstract mathematics, artificial intelligence, physics, evolution, and other subjects that happen at a different scale from normal everyday life. When I was 17, I thought I understood Quantum Mechanics just because I’d read A Brief History of Time and A Universe in a Nut Shell… Boy was I wrong!
On LessWrong, we are usually discussing subjects that are way beyond the domain of normal human experience, so we tend to be overconfident in our understanding of these subjects… but part of the reason for this overconfidence is that we do tend to be correct about most of the things we encounter within the confines of routine experience.
I was reading your comment, and when I thought about always betting a dollar, my brain went, “That’s a good idea!”
So I asked my brain, “What memory are you accessing that makes you think that’s a good idea?”
And my brain replied, “Remember that CFAR reading list you’re going through? Yeah, that one.”
So I went to my bookshelf, got out Dan Ariely’s Predictably Irrational, and started paging through it.
Professor Ariely had several insights that helped me understand why actually using money seemed like such a good idea:
Interacting within market norms makes you do a cost-benefit analysis. Professor Ariely discusses the difference between social norms and market norms in chapter 4. Social norms govern interactions that don’t involve money (favors for a friend), and market norms govern interactions that do (costs and benefits). The professor did an experiment in which he had people drag circles into a square on a computer screen (judging their productivity by how many times they did this in a set period of time). He gave one group of such participants an explicitly stated “50-cent snickers bar” and the other a “five-dollar box of Godiva chocolates.” As it turns out, the results were identical to a previous experiment in which the same amounts of direct cash were used. Professor Ariely concludes, “These results show that for market norms to emerge, it is sufficient to mention money.” In other words, Professor Ariely’s research supports your first (4.) - “A dollar feels more important than it actually is...” This is the case because as soon as money enters the picture, so do market norms.
Money makes us honest. In chapter 14, aptly titled “Why Dealing With Cash Makes Us More Honest,” Professor Ariely explains an experiment he conducted in the MIT cafeteria. Students were given a sheet of 20 math problems to solve in five minutes. The control group was to have their solutions checked, and then were given 50 cents per correct answer. A second group was instructed to tear their paper apart, and then tell the experimenter how many questions they got correct (allowing them to cheat). They were then paid 50 cents for every correct answer they claimed. Lastly, a third group was allowed to cheat similarly to the second group, except that when they gave one experimenter their score, they were given tokens, which were traded in immediately thereafter for cash through a second experimenter. The results: A) The control group solved an average of 3.5 questions correctly. B) The second group, who cheated for cash, claimed an average of 6.2 correct solutions. C) The third group, who cheated for tokens, claimed an average of 9.4 correct solutions. Simply put, when actual, physical money was removed from the subjects’ thought process by a token and a few seconds, the amount of cheating more than doubled, from 2.7 to 5.9.
In short, using money to back a prediction a) forces us to think analytically, and b) keeps us honest.
Thank you for the idea. Now I just need to find an ATM to get some ones...
This experiment does not prove that money keeps people more honest than absence of money, but more honest than token exchangeable for money. If a control group was allowed to cheat without receiving money at all they might (my prediction and I would bet a dollar on it if I didn’t use Euros) cheat even less. Then, the hypothesis “money keeps us honest” would be disproved.
I think I remember described set of experiments correctly and at least in some of them control group was definitely allowed to cheat—there were no difference in the way people turned in their results (shredding questionare and submitting only purported result on different sheet)
What would you do if he wrote and directed it? Neither? If there was more than one writer (which he was one of), or more than one director (of which he was one)?
Although there is a weight in the dollar, I think there is also another reason why people take it more seriously. People adjust their believe according to what other people believe and their confidence level. Therefore, when you propose a bet, even only for a dollar, you are showing a high confidence level and this decrease their confidence level. As a result, system 2 kicks in and they will be > [forced] to evaluate honestly.