Here is a stupid one: Detective stories. Like Encyclopedia Brown, but subtler. And with false leads. I don’t think normal mass-market detective stories would work, because they may try to deliberately choose an irrational answer to surprise you. But special ones written by rationalists for rationalists could be a fun distraction if nothing else.
That still has the problem that it doesn’t test for lack of bias, but for having bias that matches that of the people who wrote the stories. I suggest instead using real cases—and not taken from the media, because that means selection bias, but taking all the cases from the files of a particular police department during a particular span of time.
What true stories can we use besides police cases? (Also, note that in this case you’re only testing for being as smart as the police or making the same judgments as the jury—even using cases with a confession may get you false confessions.)
Point. Still, we’ve been recording lots of different kinds of events for a long time. Off the top of my head, other kinds of historical data that could be useful here:
Medical cases,
minor scientific controversies,
engineering projects,
battles,
the stock market,
markets in general,
expeditions.
Here is a stupid one: Detective stories. Like Encyclopedia Brown, but subtler. And with false leads. I don’t think normal mass-market detective stories would work, because they may try to deliberately choose an irrational answer to surprise you. But special ones written by rationalists for rationalists could be a fun distraction if nothing else.
That still has the problem that it doesn’t test for lack of bias, but for having bias that matches that of the people who wrote the stories. I suggest instead using real cases—and not taken from the media, because that means selection bias, but taking all the cases from the files of a particular police department during a particular span of time.
What true stories can we use besides police cases? (Also, note that in this case you’re only testing for being as smart as the police or making the same judgments as the jury—even using cases with a confession may get you false confessions.)
You can take cases with enough evidence to overdetermine the result, and then subtract pieces.
Point. Still, we’ve been recording lots of different kinds of events for a long time. Off the top of my head, other kinds of historical data that could be useful here:
Medical cases, minor scientific controversies, engineering projects, battles, the stock market, markets in general, expeditions.