I also had too-strong priors and “expert” ideas to be properly fox-like in my predictions, and not quick enough to update about how things were actually going based on the data. Because I was slow to move from the base-rate, I underestimated the severity of COVID-19 for too long. I’m unsure how to fix that, since most of the time it’s the right move, and paying attention to every new event is very expensive in terms of mental energy. (Suggestions welcome!)
Bottom line: in order to outperform base-rate, somebody somewhere has to do the expensive updates. No way around that. So the options are (a) do the expensive updates yourself, (b) give up and go with base rate, (c) find someone else who’s doing the updates. (c) is the obvious answer, but bear in mind that recognizing real expertise is Hard, and you’ve already noticed just how questionable the “expertise” of many supposed experts actually is.
In the easy case, the nominal experts are in a position where we get reasonably-frequent feedback on their performance (which obviously is usually not the case for rare events). In principle, one might get around that by an expert specializing in noticing rare events across domains.
Thank you—and I strongly endorse this answer. And now that you point this out, I realize that it should have been clear. I have speculated in the past that a large part of the value of Superforecasting is that there are people actually motivated to investigate and do the expensive updating I have also said that I’m unsure how worthwhile it is to pay for the time of the types of people who can superforecast. This seems like a clear case where it is worthwhile, if only it worked.
Given that, I think there’s a strong case that we need large rewards for early correct updates away from consensus, especially for very rare events. (In a case like COVID, the value of faster information is in the tens or hundreds of billions of dollars. A tiny fraction of that would be more than enough.) But the typical time-weighted forecast scores don’t account for heterogeneous update costs or give sufficient reward to figuring it out a day sooner than the average—though metaculus’s score and the scoring Ozzie Gooen has looked at are trying to do this better. This seems very worth more consideration.
Bottom line: in order to outperform base-rate, somebody somewhere has to do the expensive updates. No way around that. So the options are (a) do the expensive updates yourself, (b) give up and go with base rate, (c) find someone else who’s doing the updates. (c) is the obvious answer, but bear in mind that recognizing real expertise is Hard, and you’ve already noticed just how questionable the “expertise” of many supposed experts actually is.
In the easy case, the nominal experts are in a position where we get reasonably-frequent feedback on their performance (which obviously is usually not the case for rare events). In principle, one might get around that by an expert specializing in noticing rare events across domains.
Thank you—and I strongly endorse this answer. And now that you point this out, I realize that it should have been clear. I have speculated in the past that a large part of the value of Superforecasting is that there are people actually motivated to investigate and do the expensive updating I have also said that I’m unsure how worthwhile it is to pay for the time of the types of people who can superforecast. This seems like a clear case where it is worthwhile, if only it worked.
Given that, I think there’s a strong case that we need large rewards for early correct updates away from consensus, especially for very rare events. (In a case like COVID, the value of faster information is in the tens or hundreds of billions of dollars. A tiny fraction of that would be more than enough.) But the typical time-weighted forecast scores don’t account for heterogeneous update costs or give sufficient reward to figuring it out a day sooner than the average—though metaculus’s score and the scoring Ozzie Gooen has looked at are trying to do this better. This seems very worth more consideration.