Incentivizing strong judge performance seems difficult. The default outcome I’ve seen from committees and panels is that everyone on them half-asses their job, because they rarely have stake in the outcome being good. Even if someone cares about LessWrong, that is not the same as being generally held personally responsible for it, and having your salary and broader reputation depend on how well LessWrong is going.
Couldn’t prediction markets solve this? Make one for decisions by judges asking whether you’d agree with them. After some time, randomly choose to investigate one such market and resolve it. Perhaps make pay for judges conditional on the market predicting high agreement w/ your judgement. Of course, there’s liquidity issues, but: 1) these events are “hot” and would attract lots of betters, 2) you probably don’t need sub 5% accuracy here, 3) don’t make resolution more than like a year out or so out.
Couldn’t prediction markets solve this? Make one for decisions by judges asking whether you’d agree with them. After some time, randomly choose to investigate one such market and resolve it. Perhaps make pay for judges conditional on the market predicting high agreement w/ your judgement. Of course, there’s liquidity issues, but: 1) these events are “hot” and would attract lots of betters, 2) you probably don’t need sub 5% accuracy here, 3) don’t make resolution more than like a year out or so out.