Suppose several people are guessing a number, and then find an estimate to see who is right.
The super-common protocol is: whoever is closest, wins.
This protocol is really bad. If there are three people, and I guess 50, then the other two people can guess 51 and 49. This means I’ll almost certainly lose. Unless it’s within 1⁄2 of 50, one of the other guesses will be closer.
There are lots of ways to fix this protocol. However, most of them suffer from too much added complexity. For example, squared error incentivises everyone to guess their expected value (mean). However, people don’t generally want to calculate squares, and if they did, they’d still feel like the lowest squared error was the winner (which amounts to the usual protocol).
Or, we could give confidence intervals. But how should they be scored?
My question is this: what are some ways to play this game that combine simplicity with good incentives?
If you must find a single winner, allow people to change their guess after hearing the later ones, continuing until everyone is happy with their guess (and implicit range). Thus, if player 1 was thinking “somewhere around 50” they can now pick between “exactly 50″ “lower” or “higher;” and there will be an auction to determine how far below 50 someone stops (or at what point the “51” player drops down, and maybe gets “overcut”
You could also use something like the “name that tune” algorithm:
The first guesser picks a range of values
The second guesser picks a range of values that either A. Does not overlap with the first guesses OR B. Covers a different number of values as the first guesser
Each subsequent guesser has the same options A. Make a guess that doesn’t overlap anyone B. Make a guess with a different number of values as anyone who is overlapped
Allow people to revise their ranges, as above.
Then, whoever has the smallest range that contains the true value wins.
This answer is great because it takes the problem with the initial game (one person gets to update and the other doesn’t) and returns the symmetry by allowing both players to update. The end result shows who is better at Aumann updating and should get you closer to the real answer.
If you’d rather know who has the best private beliefs to start with, you can resolve the asymmetry in the other direction and make everyone commit to their numbers before hearing anyone else’s. This adds a slight bit of complexity if you can’t trust the competitors to be honest, but it’s easily solved by either paper/pencil or everyone texting their answer to the person who is going to keep their phone in their pocket and say their answer first.
Two good suggestions!
I particularly like the first suggestion as an extremely simple and lightweight modification to the standard procedure. OTOH, it does incentivise rather dishonest reports (since players still want to strategically take under-covered ranges).
The second proposal is pretty good as well, although to avoid the complexity of “do not overlap OR cover a different number of values” I might prefer to simply allow ties.
The problem with the existing protocol is that it forces the choice of a single winner. If multiple players are all basically right, the protocol you describe forces them into a deathmatch because only one player can be “the winner”.
(Another problem with the existing protocol is that it has some players making their predictions “before” others, in a way that is visible to the others.)
Here’s a better protocol: everyone makes their prediction at the same time without seeing anyone else’s prediction. If someone is off by X units then their score for that round is 1/(X+1). For best results, play several rounds and compute the average score.
You might also be interested in Wits And Wagers, which is the “everyone predicts a number” activity made into a six-player board game. I’ve played it. It’s pretty fun.
What they said. Adjust the protocol to allow everyone equally close to be the “winner”
As a simple protocol, take the true number, and check at each order of magnitude:
0. Anyone who got it exact wins
Anyone within 90% − 110% of the true value wins. If that’s nobody
Anyone within 50% − 200% of the true value wins. If that’s still nobody
Anyone within 10% − 1000% of the true value wins.
If that’s nobody, then everyone loses.
Everyone settling on an answer before anyone speaks is a good norm in general to avoid anchoring, in many settings.
However, when playing with non-rationalists, I feel like one would need paper in order to implement it in a trustworthy way, which makes me think it’s not going to be popular for that use-case.
It seems like the most straightforward simple fix is for everyone to guess simultaneously. Then people can’t just grab big ranges because they are big. E.g. the player who guesses 49 in your example would have to have actually had a lower estimate for the value rather than merely cutting you off at the knees. I haven’t studied in depth but I think the incentives (if each person is trying to account what the others, N-layers deep) are to just guess your true prediction.
I suspect that the incentive is far from that; instead, the strategic response is to randomize somewhat, so that you expect to land in a relatively unclaimed region.
However, this strategy is counter-intuitive enough that humans may just be honest instead, relying on differences in beliefs to inject some randomization for you.
When played as a game, the purpose is rarely for the organizer to get a good estimate. It’s either for entertainment or for point-scoring by the participants. As such, the bracketing strategy is just part of the fun. If you want to “fix” it, just make it simultaneous—you get better information about independent estimates, and you remove (some of) the strategic adjustment of guesses.
You could also give up the “one winner” idea—scoring based on distance from actual (or better, by impact of error, which can vary by topic), without caring whether someone else was closer, would incent true estimate submission.
Finally, why not just set up a betting market? Let participants weight their estimates by strength of information.
If you’re just doing this occasionally without recordkeeping, then it seems convenient to have the game result in “winners” rather than a more fine-grained score. But it could be fine to sometimes have multiple winners, or zero winners. Here’s a simple protocol that does that:
The person who asks the question also defines what counts as “winning”. e.g. “What’s the value of such-and-such? Can anybody get it within 10%?” Then everyone guesses simultaneously, and all the people whose guesses are within 10% of the true value are “winners”.
(“Simultaneous” guessing can mean that first everyone comes up with their guess in their head, and then they take turns saying them out loud while on the honor system to not change their guess.)
Slightly more complicated, the asker could propose 2 standards of winning. “When did X happen? Grand prize if you guess the exact year, honorable mention if you get it within 5 years.” Then if anyone guesses the exact year they’re the big winner(s) and the people who get it within 5 years get the lesser glow of “honorable mention”. And if no one guesses the exact year then the people who get it within 5 years feel more like winners.
If you continue farther in this direction you could get to one of Ericf’s proposals. I think my version has lower barriers to entry, while Ericf’s version could work better among people who use it regularly.
Get each player to assign a probability distribution over answers. Starting with an equal prior over players answers, update it based on the observation. Sample from the posterior. (So if Alice assigned twice as much prob as Bob to the correct outcome, then Alice is twice as likely to win. )
A relatively simply way to do this would be rather than CI, you give “location and scale” and are scored according to:
∣∣∣result−locationscale∣∣∣Can you clarify (possibly by giving an example)? Are players are trying to minimize their score as calculated by this method?
And if so, is there any incentive to not just pick a huge number for the scale to minimize that way?
Here is a proposal. Not sure if I like it better or worse than my other one.
In all cases, add infinitesimal noise to everyones votes, and the final answer, to avoid ties.
If there are an odd number 2n+1≥3 of people, have everyone submit a n2n+1−n+12n+1 confidence interval. Alternate (choosing randomly which to do first) taking the unlabeled person with the lowest remaining lower bound, and labeling them “below”, and taking the unlabeled person with the highest remaining upper bound, and labeling them “above.” When one person remains unlabeled, they are given the region (r0,r1), where r0 is the greatest lower bound among people labeled “below,” and r1 is the least upper bound among people labeled “above.” Recursively distribute the remaining region less than r0 across people labeled below and the remaining region greater than r1 across people labeled above.
If there are an even number 2n≥2 of people, have everyone name their median guess, let r be the average of the two inner-most guesses, and recursively distribute the region less than r to the people who gave guesses less than r, and the region greater than r to the people who gave guesses greater than r.
If there is 1 person, they get the entire region.
This is my initial proposal, but I am going to think about it more:
There are n people. Everyone privately names a number, where the prompt is “What number do you think the answer has probability 1/n of being below.” Lowest bidder gets the territory (−∞,b1), where b1 is the second lowest bid.
The remaining n−1 people divide up the [b1,∞) using the same process, with the prompt “What number do you think the answer has probability 1/(n−1) of being below conditioned on being at least b1. Lowest bidder getting the territory [b1,b2), where b2 is the second lowest bid.
Continue this pattern until there is only one person left, who gets the remaining region.
This is not symmetric with respect to negating the answers, but it does have the property that everyone has a strategy (honest reporting) that guarantees that they win with subjective probability at least 1/n.
I am not sure that the second price part of this algorithm is actually doing anything useful, but it feels more fair, and closer to incentivizing honest reporting than the first price version. I am not actually sure whether it is better to leave at the beginning or end if everyone reports honestly for the second price version. The first price version is equivalent to:
Move the knife slowly across the cake from left to right. Anyone can say “cut” at any time and get the piece to the left of the knife and leave the game. Repeat.
It is not exactly equivalent, because people get to see each other’s bids from the previous round.
Everyone has to name a guess (mean) and range (standard deviation) of a normal distribution. Whoever’s pdf takes the largest value at the true answer wins. Bonus: you may opt to invertibly transform the input first in one of several acceptable ways, most notably taking the logarithm. Now take that and simplify it. ;)
Something like that could work, but it seems like you would still need to have a rule that you must guess before you know the other players guesses.
Otherwise, player 2 could simply guess the same mean as player 1 - with a slightly larger standard deviation—and have a PDF that takes a higher value everywhere except for a very small interval around the mean itself.
Alternatively, if 3 players all guessed the same standard deviation, and the means they guessed were 49, 50, and 51, then we would have the same problem that the opening post mentions in the first place.
This seems fairly nice, except that actually doing the math is far too demanding for most settings where I’d play this game.
Everyone guesses a number (could be sequentially or simultaneously, but duplicate guesses are allowed).
Anyone who guessed the correct value in the previous step wins.
If no guesses were exactly correct, everyone guesses a new number and repeat until one or more players guess correctly.
This only works for reasonably small discrete ranges. For continuous or very large ranges, modify step 2 so that anyone who guesses within some distance of the true value wins.
Is this for a one-shot game or are you doing this over many iterations with players getting some number of points each round?
One simple method (if you are doing multiple rounds) is to rank players each round (Closest=1st, Second Closest=2nd, etc) and assign points as follows:
Points = Number of Players—Rank
So say there are 3 players who guess as follows:
And say the actual number is 52.
So their ranks for that round would be:
And their scores would be:
I think this works better if you are calculating a winner over many rounds, so that there is a new ranking and new awarding of points on each round. The same is true of least squared error, which you mention, and most of the other methods of incentivizing players to try to guess the mean expected value.
I could also think of other ways to incentivize this, and to use confidence intervals, but they all add complexity to the points calculations.
Alas, it’s usually one-shot.
In that case, the options are really limited and the main simple ideas for that (eg: guess before you know other player’s guesses) have been mentioned already.
One other simple method for one-shot number games I can think of is:
Automatic Interval Equalization:
When all players guesses are known, you take the two players whose guesses are closest and calculate half the difference between them. That amount is the allowable error, and each player’s interval is his or her guess, plus or minus that allowable error.
You win if and only if the answer is in your interval.
Example:
This would result in at most one winner (unless the answer is half way between the 2 closest guesses). Everyone’s winning interval would be the same size and none would overlap. And nobody would have an incentive to guess near someone else’s (stated or expected) guess, unless they thought the answer was actually close to that.
However, it has the disadvantage that a lot of such contests would end up with no winner.