I think the issue is that creating an incentive system where people are rewarded for being good at an artificial game that has very little connection to their real world cericumstances, isn’t going to tell us anything very interesting about how rational people are in the real world, under their real constraints.
I have a friend who for a while was very enthused about calibration training, and at one point he even got a group of us from the local meetup + phil hazeldon to do a group exercise using a program he wrote to score our calibration on numeric questions drawn from wikipedia. The thing is that while I learned from this to be way less confident about my guesses—which improves rationality, it is actually, for the reasons specified, useless to create 90% confidence intervals about making important real world decisions.
Should I try training for a new career? The true 90% confidence interval on any difficult to pursue idea that I am seriously considering almost certainly includes ‘you won’t succeed, and the time you spend will be a complete waste’ and ‘you’ll do really well, and it will seem like an awesome decision in retrospect’.
Elon Musk estimated a 10% chance of success for both Tesla and SpaceX. Those might have been good estimates.
Peter Thiel talks about how one of the reasons that the PayPal mafia is so successful is that they all learned that success is possible but really hard. If you pursue a very difficult idea and know you have a 10% chance of success you really have to give it all and know that if you slack you won’t succeed.
I think the issue is that creating an incentive system where people are rewarded for being good at an artificial game that has very little connection to their real world cericumstances, isn’t going to tell us anything very interesting about how rational people are in the real world, under their real constraints.
I have a friend who for a while was very enthused about calibration training, and at one point he even got a group of us from the local meetup + phil hazeldon to do a group exercise using a program he wrote to score our calibration on numeric questions drawn from wikipedia. The thing is that while I learned from this to be way less confident about my guesses—which improves rationality, it is actually, for the reasons specified, useless to create 90% confidence intervals about making important real world decisions.
Should I try training for a new career? The true 90% confidence interval on any difficult to pursue idea that I am seriously considering almost certainly includes ‘you won’t succeed, and the time you spend will be a complete waste’ and ‘you’ll do really well, and it will seem like an awesome decision in retrospect’.
Elon Musk estimated a 10% chance of success for both Tesla and SpaceX. Those might have been good estimates.
Peter Thiel talks about how one of the reasons that the PayPal mafia is so successful is that they all learned that success is possible but really hard. If you pursue a very difficult idea and know you have a 10% chance of success you really have to give it all and know that if you slack you won’t succeed.