I feel like this post made the point “You can come up with many plausible outside views for a given question”.
But it didn’t really give me what I wanted when I clicked the title: discussion of how to choose between outside views (whether those be concrete heuristics or philosophical arguments).
I’d be very curious to read some of your battles stories or case studies on this from your superforecasting years.
I tried to kind-of do this with the examples here. Unfortunately, I don’t actually have a clear recollection of what I did for forecasting many questions, as it’s been a long time since the original tournament. And for more recent questions, I often comment of metaculus—but if longer analyses seem worthwhile, maybe that should be a post to accompany, say, my 2021 yearly predictions. (I’m naively expecting to have free time by then.)
But for most questions, the outside view is relatively easy. That doesn’t mean it’s the standard “use a reference class,” since as I said, models dictate that. My favorite example of that is asset price forcasts, where I remember that some super-forecasters were building their own models of asset prices and probability of movement by a certain amount in a given time period from historical data, and I was just forecasting the implicit price distribution given by options prices, and absolutely dominating brier scores for those questions. (Occasionally I had very slight modifications to reflect my inside view of surprises outside that model, like stock splits and dividends, where actually modelling it correctly was annoying and not worthwhile.)
For other questions, like forecasting life-spans of dictators, the answer is fundamentally hard, and I don’t think reference classes are nearly as valuable. And for COVID, I’ve written about my very early expectations—but maybe you think that a follow-up on why superforecasters mostly disagreed with my forecasts / I modeled things differently than them over the past 3-4 months would be interesting and useful. (I would need to check what amount of that type of discussion I can discuss publicly.)
Edit to add: There are also some interesting things to discuss around epistemic superiority, and how to deal with a relative lack of expertise in choosing between expert views or in deciding how and when it makes sense to disagree as a “general expert” with forecasting expertise. That’s a bit more philosophical, but I’m hoping to discuss related issues in a paper on elicitation I am writing.
And for COVID, I’ve written about my very early expectations—but maybe you think that a follow-up on why superforecasters mostly disagreed with my forecasts / I modeled things differently than them over the past 3-4 months would be interesting and useful
I feel like this post made the point “You can come up with many plausible outside views for a given question”.
But it didn’t really give me what I wanted when I clicked the title: discussion of how to choose between outside views (whether those be concrete heuristics or philosophical arguments).
I’d be very curious to read some of your battles stories or case studies on this from your superforecasting years.
I tried to kind-of do this with the examples here. Unfortunately, I don’t actually have a clear recollection of what I did for forecasting many questions, as it’s been a long time since the original tournament. And for more recent questions, I often comment of metaculus—but if longer analyses seem worthwhile, maybe that should be a post to accompany, say, my 2021 yearly predictions. (I’m naively expecting to have free time by then.)
But for most questions, the outside view is relatively easy. That doesn’t mean it’s the standard “use a reference class,” since as I said, models dictate that. My favorite example of that is asset price forcasts, where I remember that some super-forecasters were building their own models of asset prices and probability of movement by a certain amount in a given time period from historical data, and I was just forecasting the implicit price distribution given by options prices, and absolutely dominating brier scores for those questions. (Occasionally I had very slight modifications to reflect my inside view of surprises outside that model, like stock splits and dividends, where actually modelling it correctly was annoying and not worthwhile.)
For other questions, like forecasting life-spans of dictators, the answer is fundamentally hard, and I don’t think reference classes are nearly as valuable. And for COVID, I’ve written about my very early expectations—but maybe you think that a follow-up on why superforecasters mostly disagreed with my forecasts / I modeled things differently than them over the past 3-4 months would be interesting and useful. (I would need to check what amount of that type of discussion I can discuss publicly.)
Edit to add: There are also some interesting things to discuss around epistemic superiority, and how to deal with a relative lack of expertise in choosing between expert views or in deciding how and when it makes sense to disagree as a “general expert” with forecasting expertise. That’s a bit more philosophical, but I’m hoping to discuss related issues in a paper on elicitation I am writing.
I’d be interested in this.