Eliezer also wrote an interesting comment on the EA Forum crosspost of this article, copying it here for convenience:
I worriedly predict that anyone who followed your advice here would just switch to describing whatever they’re doing as “reference class forecasting” since this captures the key dynamic that makes describing what they’re doing as “outside viewing” appealing: namely, they get to pick a choice of “reference class” whose samples yield the answer they want, claim that their point is in the reference class, and then claiming that what they’re doing is what superforecasters do and what Philip Tetlock told them to do and super epistemically virtuous and anyone who argues with them gets all the burden of proof and is probably a bad person but we get to virtuously listen to them and then reject them for having used the “inside view”.
My own take: Rule One of invoking “the outside view” or “reference class forecasting” is that if a point is more dissimilar to examples in your choice of “reference class” than the examples in the “reference class” are dissimilar to each other, what you’re doing is “analogy”, not “outside viewing”.
All those experimental results on people doing well by using the outside view are results on people drawing a new sample from the same bag as previous samples. Not “arguably the same bag” or “well it’s the same bag if you look at this way”, really actually the same bag: how late you’ll be getting Christmas presents this year, based on how late you were in previous years. Superforecasters doing well by extrapolating are extrapolating a time-series over 20 years, which was a straight line over those 20 years, to another 5 years out along the same line with the same error bars, and then using that as the baseline for further adjustments with due epistemic humility about how sometimes straight lines just get interrupted some year. Not by them picking a class of 5 “relevant” historical events that all had the same outcome, and arguing that some 6th historical event goes in the same class and will have that same outcome.
Eliezer also wrote an interesting comment on the EA Forum crosspost of this article, copying it here for convenience: