In short, I recommend against including this post in the 2020 review.
Reasons against inclusion
Contained large mistakes in the past, might still contain mistakes (I don’t know of any)
I fixed the last mistakes I know of two months ago
It’s hard to audit because of the programming language it’s written in
Didn’t quite reach its goal
I wanted to be able to predict the decrease in ability to forecast long-out events, but the brier scores are outside the 0-1 range for long ranges (>1 yr), which shouldn’t be the case if we’re predicting it
To solve this, I would probably need to perform a similar analysis in logspace (probably also with log scores instead of Brier scores?)
No literature review
I don’t know whether there has been any similar analysis in academia, the text by Muehlhauser doesn’t mention any (although it might be worth examining Millner & Heyen 2021, and of course there has also been the work by Charles Dillon at Rethink Priorities)
Too technical for the books
Contains a lot of code in a programming language ~100 people know, so not useful to the readers
Leave the code out or make it so beautiful that nobody really cares
Uses a very obscure programming language
It doesn’t use the PredictionBook API
Reasons for inclusion:
As far as I know, this is the first post to look at this question
Even if it isn’t perfect, it probably doesn’t contain any fundamental errors
The question itself seems pretty important: How far can we look into the future, and how well?
(I am the author of this piece)
In short, I recommend against including this post in the 2020 review.
Reasons against inclusion
Contained large mistakes in the past, might still contain mistakes (I don’t know of any)
I fixed the last mistakes I know of two months ago
It’s hard to audit because of the programming language it’s written in
Didn’t quite reach its goal
I wanted to be able to predict the decrease in ability to forecast long-out events, but the brier scores are outside the 0-1 range for long ranges (>1 yr), which shouldn’t be the case if we’re predicting it
To solve this, I would probably need to perform a similar analysis in logspace (probably also with log scores instead of Brier scores?)
No literature review
I don’t know whether there has been any similar analysis in academia, the text by Muehlhauser doesn’t mention any (although it might be worth examining Millner & Heyen 2021, and of course there has also been the work by Charles Dillon at Rethink Priorities)
Too technical for the books
Contains a lot of code in a programming language ~100 people know, so not useful to the readers
Leave the code out or make it so beautiful that nobody really cares
Uses a very obscure programming language
It doesn’t use the PredictionBook API
Reasons for inclusion:
As far as I know, this is the first post to look at this question
Even if it isn’t perfect, it probably doesn’t contain any fundamental errors
The question itself seems pretty important: How far can we look into the future, and how well?
Was the basis for at least one other investigation into the topic
So one might argue that it was slightly influential
It is more empirical than a lot of work on lesswrong, which I think is good
Filling the frame provided by Muehlhauser
Future plans
I plan to continue working on this post, hopefully
obtaining the Metaculus dataset and running the analysis on it
rewriting the code in Python or R
performing linear regressions in logspace (?) so that there is no prediction of >1 Brier scores for some predictions
adding analysis for continuous predictions using the scoring rules described in Greenberg 2020
adding a literature review
analyzing further datasets, perhaps using data from Metaforecast
As expected, I found another mistake: the conclusion was outdated (referring to old results). This has been fixed.