Harold Washington Library, South State Street, Chicago, IL, USA
Contact: joshua.aaron.becker@gmail.com
[ 6th Floor North Study Room // BYO snacks ]
We will read two (very short) academic papers describing how Bayes rule can be used to extract objective truth from fallibly human survey data. The first paper is fairly technical, so I (Joshua) recommend reading only certain selections (defined below, with a link to an annotated PDF) that give a high-level conceptual overview. The second paper is very short and conceptual. While we will get into the mathematical proof of these concepts—otherwise, how could we believe they really work?--Joshua has volunteered to walk us through that process, and promises that the math is really quite simple, if mind-bending.
The first paper, published in 2004, defines the “Bayesian Truth Serum.” This serum is a scoring method that incentivizes people toward honest reporting in subjective data, such as: is this art good? In one more pragmatic example, the author discusses a survey asking people the odds that Humanity will survive past the year 2100. This question is important, but we have no way of telling whether people are giving their true belief. The method proposed in this paper, through a clever form of Bayesian witchcraft, ensures that honest reporting is a Bayesian Nash Equilibrium.
The second paper, published in 2017, is basically just magic: it shows how we can use a similar method to extract objective truth from survey responses where there is some possibility of defining accuracy, building on “the wisdom of crowds” principle. For example, we could use this method to ask: will Donald Trump be re-elected in 2020? Both methods share the common approach of asking people both to provide their own answer as well as guessing what answers other people will provide. It turns out that for objective truth, the “surprisingly popular” answer is more likely to be correct.
all of the first page (p.462), through to the last full paragraph of the first column on the second page (p.463)
second page (p.463) column 2, starting at “To see how this works..” through to the first full paragraph of column 3 (stop at the comment on the SOM text)
third page (p.464), full paragraphs in first column
Bayesian Witchcraft: Finding Objective Truth in Survey Data
[ 6th Floor North Study Room // BYO snacks ]
We will read two (very short) academic papers describing how Bayes rule can be used to extract objective truth from fallibly human survey data. The first paper is fairly technical, so I (Joshua) recommend reading only certain selections (defined below, with a link to an annotated PDF) that give a high-level conceptual overview. The second paper is very short and conceptual. While we will get into the mathematical proof of these concepts—otherwise, how could we believe they really work?--Joshua has volunteered to walk us through that process, and promises that the math is really quite simple, if mind-bending.
The first paper, published in 2004, defines the “Bayesian Truth Serum.” This serum is a scoring method that incentivizes people toward honest reporting in subjective data, such as: is this art good? In one more pragmatic example, the author discusses a survey asking people the odds that Humanity will survive past the year 2100. This question is important, but we have no way of telling whether people are giving their true belief. The method proposed in this paper, through a clever form of Bayesian witchcraft, ensures that honest reporting is a Bayesian Nash Equilibrium.
The second paper, published in 2017, is basically just magic: it shows how we can use a similar method to extract objective truth from survey responses where there is some possibility of defining accuracy, building on “the wisdom of crowds” principle. For example, we could use this method to ask: will Donald Trump be re-elected in 2020? Both methods share the common approach of asking people both to provide their own answer as well as guessing what answers other people will provide. It turns out that for objective truth, the “surprisingly popular” answer is more likely to be correct.
Link to PDFs:
1. http://tiny.cc/bayesian-truth-serum
2. http://tiny.cc/surprisingly-popular (annotated)
Reading selections for the Bayesian truth serum:
all of the first page (p.462), through to the last full paragraph of the first column on the second page (p.463)
second page (p.463) column 2, starting at “To see how this works..” through to the first full paragraph of column 3 (stop at the comment on the SOM text)
third page (p.464), full paragraphs in first column
fourth page (p.465) first full paragraph to end