Maybe I misunderstand how this works, but if a correct prediction is made by 1 genius and 100 cranks, making this prediction should still be treated as a smart thing. Because:
punishing the right answer just feels wrong;
you are not supposed to perfectly distinguish between geniuses and cranks based on one prediction;
if you evaluate many different predictions, then the crank will randomly succeed at one and fail at hundred, resulting in a negative score, while the genius will succeed at many and fail at a few, resulting in a positive score, so now everything works as expected.
It seems like a base-rate fallacy. Assuming that geniuses are generally better at predictions than cranks, the explanation why the difficult correct prediction was made by 1 genius and 100 cranks is that the population contains maybe 10 geniuses and 100 000 cranks, and on a specific hard answer, the genius has a 10% chance of success by thinking hard, and the crank has a 0.1% of success by choosing a random thing to believe.
But this means that awarding the “correctness point” to the 1 genius and 100 cranks is okay in long term, because the genius will keep collecting points, but for the crank it was the only point earned for a long time.
I think your understanding is generally correct. The failure case I see is where people say “this problem was really really really hard, instead of one point, I’m going to award one thousand correctness points to everyone who predicted it”, and then end up surprised that most of those people still turn out to be cranks.
Maybe I misunderstand how this works, but if a correct prediction is made by 1 genius and 100 cranks, making this prediction should still be treated as a smart thing. Because:
punishing the right answer just feels wrong;
you are not supposed to perfectly distinguish between geniuses and cranks based on one prediction;
if you evaluate many different predictions, then the crank will randomly succeed at one and fail at hundred, resulting in a negative score, while the genius will succeed at many and fail at a few, resulting in a positive score, so now everything works as expected.
It seems like a base-rate fallacy. Assuming that geniuses are generally better at predictions than cranks, the explanation why the difficult correct prediction was made by 1 genius and 100 cranks is that the population contains maybe 10 geniuses and 100 000 cranks, and on a specific hard answer, the genius has a 10% chance of success by thinking hard, and the crank has a 0.1% of success by choosing a random thing to believe.
But this means that awarding the “correctness point” to the 1 genius and 100 cranks is okay in long term, because the genius will keep collecting points, but for the crank it was the only point earned for a long time.
I think your understanding is generally correct. The failure case I see is where people say “this problem was really really really hard, instead of one point, I’m going to award one thousand correctness points to everyone who predicted it”, and then end up surprised that most of those people still turn out to be cranks.