I think that viewing it as a competition to place highly on the leaderboards is misleading, and perhaps even damaging.
I’d think the better framing for metaculus points is that they are like money—you are being paid to predict, on net, and getting more money is better. The fact that the leaderboard has someone with a billion points, because they have been participating for years, is kind-of irrelevant, and misleading.
In fact, I’d like to see metaculus points actually be convertible to money at some point in some form—and yes, this would require a net cost (in dollars) to post a new question, and have the pot of money divided proportionate to the total points gained on the question—with negative points coming out of a users’ balance. (And this would do a far better job aligning incentives on questions than the current leaderboard system, since for a leaderboard system, proper scoring rules for points are not actually incentive compatible.)
The fact that the leaderboard has someone with a billion points, because they have been participating for years, is kind-of irrelevant, and misleading.
There are many leaderboards, including ones that only consider questions that opened recently. Or tournaments with a distinct start and end date.
(And this would do a far better job aligning incentives on questions than the current leaderboard system, since for a leaderboard system, proper scoring rules for points are not actually incentive compatible.)
This is true, but you can create leaderboards that minimize the incentive to use variance-increasing strategies (or variance-decreasing ones if you’re in the lead). (Basically just include a lot of questions so that variance-increasing strategies will most likely backfire, and then have gradually increasing payouts for better rankings.)
I agree that what you describe sounds ideal, and maybe it makes sense for Metaculists to think of the points in that way. For making it a reality, I worry that it would cost a lot. (And you’d need a solution against the problem that everyone who wants a few extra dollars could create an account to predict the community median on every question to get some fraction of the total prize pool for just that.)
If points could be converted to money enough to motivate real predictions, I would expect a flood of people who do nothing but information cascade to bank points, and it’s not obvious what to do about that. As it is, it felt (to me) like there was a tension between ‘score points’ and ‘make good predictions or at least don’t make noise predictions’ and that felt like a dealbreaker.
I agree that actually offering money would require incentives to avoid, essentially, sybil attacks. But making sure people don’t make “noise predictions” isn’t a useful goal—those noise predictions don’t really affect the overall metaculus prediction much, since it weights past accuracy.
I think that viewing it as a competition to place highly on the leaderboards is misleading, and perhaps even damaging.
I’d think the better framing for metaculus points is that they are like money—you are being paid to predict, on net, and getting more money is better. The fact that the leaderboard has someone with a billion points, because they have been participating for years, is kind-of irrelevant, and misleading.
In fact, I’d like to see metaculus points actually be convertible to money at some point in some form—and yes, this would require a net cost (in dollars) to post a new question, and have the pot of money divided proportionate to the total points gained on the question—with negative points coming out of a users’ balance. (And this would do a far better job aligning incentives on questions than the current leaderboard system, since for a leaderboard system, proper scoring rules for points are not actually incentive compatible.)
There are many leaderboards, including ones that only consider questions that opened recently. Or tournaments with a distinct start and end date.
This is true, but you can create leaderboards that minimize the incentive to use variance-increasing strategies (or variance-decreasing ones if you’re in the lead). (Basically just include a lot of questions so that variance-increasing strategies will most likely backfire, and then have gradually increasing payouts for better rankings.)
I agree that what you describe sounds ideal, and maybe it makes sense for Metaculists to think of the points in that way. For making it a reality, I worry that it would cost a lot. (And you’d need a solution against the problem that everyone who wants a few extra dollars could create an account to predict the community median on every question to get some fraction of the total prize pool for just that.)
If points could be converted to money enough to motivate real predictions, I would expect a flood of people who do nothing but information cascade to bank points, and it’s not obvious what to do about that. As it is, it felt (to me) like there was a tension between ‘score points’ and ‘make good predictions or at least don’t make noise predictions’ and that felt like a dealbreaker.
I agree that actually offering money would require incentives to avoid, essentially, sybil attacks. But making sure people don’t make “noise predictions” isn’t a useful goal—those noise predictions don’t really affect the overall metaculus prediction much, since it weights past accuracy.