Experienced quant trader, based in Dublin. Formerly a volunteer at Rethink Priorities, where I did some forecasting research. Interested in longtermism and meta causes. More active on the EA Forum https://forum.effectivealtruism.org/users/charles_dillon
CharlesD
Karma: 79
Thanks for a great post! I have a concern about your sample sizes however.
I am looking into similar questions myself, and while reading your post I was surprised to see your Metaculus sample claimed as 45k predictions. These are not actually individual predictions, but rather the time series of community predictions, which are much less information dense, as this is just the median of the recent community predictions at that time and typically a new prediction will have a small effect on this value. I think claiming the sample size is 45k is therefore a bit misleading.
It also has the effect of linearly weighting Metaculus questions by community interest, which is not obviously a desirable method (this is mitigated by the cap on the time series length to 101, which means the effect will ultimately be small as you have 557 Metaculus questions, implying your average question must have more than 80 predictions)