[Question] Tracking accuracy of personal forecasts
I’ve been thinking how I can improve my accuracy predicting events of personal interest (e.g., “Will my landlord get the washing machine fixed within the next two weeks”, or “Will my parent die this year” for a more extreme example). Betting markets will not help me with that.
At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.
Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?
As an RFC, here’s the spreadsheet layout I have on mind:
Tags: (value 0 or 1):
Health
Finance
Interpersonal relations
...
Date of the forecast
Event (e.g., “My landlord will get the washing machine fixed within the next two weeks”). I’m planning to formulate them so that “yes” is always the desired outcome, so that it’s easy to spot if I’m reliably too optimistic or pessimistic.
Estimated probability
Deadline of the forecast
Outcome (value 0 or 1, filled after the deadline of the forecast, or when the answer is known sooner)
You might want to take a look at https://predictionbook.com/
Thank you! As I hinted in my response to Nebu above, I can see there are some personal predictions as well. I have a couple doubts about using it, though:
I wouldn’t want to publicly predict very personal events, even if it’s pseudonymous, done through VPN and marked as non-public. Just knowing that such a sensitive information is sitting on a server somewhere would make me uncomfortable.
When analysing predictions, grouping them by tags is a necessary feature for me. For example, I suspect I’m too pessimistic about my personal finance, but too optimistic about interpersonal relations. In fact, this impression is what got me started on this idea in the first place. I would consider implementing this feature in PredictionBook, but then I would want to do that through my regular github account, so my identity could be linked to my predictions even easier.
There doesn’t seem to be any way to export your predictions for a more advanced analysis.
Overall, though, this is a great project and very relevant. I’m just being very picky.
I just made predictions, a computer program that analyzes tagged predictions, public. If you’re comfortable compiling Go programs you might want to check it out. (Fair warning: the documentation and its organization hasn’t quite caught up with the program.)
Sweet, thank you! I will definitely try it out.
From a brief skim (e.g. “A Democratic candidate other than Yang to propose UBI before the second debate”, “Maduro ousted before end of 2019″, “Donald Trump and Xi Jinping to meet in March 2019”, etc.), this seems to be focused on “non-personal” (i.e. global) events, whereas my understanding is the OP is interested in tracking predictions for personal events.
Perhaps there’s some back story to this post that I missed, so forgive me if what I’m about to say has been discussed.
You might consider reading “Superforecasting: The Art and Science of Prediction,” by Philip Tetlock. Or go to the Good Judgment Project web site and watch the 5-part Superforecasting master class.
First, the question has to pass the clairvoyant test. Second, you might want to have some scheme for Bayesian updating your forecast. And then you’ll want to use Brier Scores (or something like them) to assess your accuracy.
If you know R, there’s actually a Brier score function you can use. But I can’t imagine it’s very difficult to set up in Excel.
Again, sorry if I’m stating the obvious.
Thanks. This is all very relevant. And no, there is no backstory, at least not that I shared anywhere.
Yes, I read “Superforcasting”. I didn’t know they had masterclass and will look it up. I suspect their teachings will be somewhat less useful for predictions of personal importance: different biases will be at play here. But it should be worthwhile to watch it anyway.
Actually, I think if you are going to assess your own predictions, you can afford the luxury of being a bit less specific, especially if the prediction is made in the short-term. For example, consider a made-up question:
“Will Adam be able to get back to cycling within a month [after a recent accident]?”
If Adam resumes cycling but it causes him considerable pain, I know that’s not what I intended when asking the original question. On the other hand, if Adam recovers fully but starts playing rugby instead of cycling because he discovers he enjoys it more, I know the answer to the intended question is “yes”. (The imprecise part of the question here is “be able to”, but as long as I can reliably recall the intention when writing those words, they cause no loss of precision.)
Hmm. For now I was planning to make my predictions once and forget about them until the outcome is known. I’m not sure I want to spend more effort on them, at least not so early into the project.
Brier score would be great at telling me how accurate I am, but not what mistakes I’m making, at least based on my very limited understanding of the metric. As a basic analysis method, I was planning to group my predictions by the forecast probability (e.g., 0%-10%, 10%-20%, … ranges, or maybe ranges 1pp wide at the extremes and growing exponentially towards the centre, that would probably make more sense), and simply chart them grouped by tag. I must admit, my knowledge of statics has always been very poor, so I’m sure there is some better analysis/visualisation methodology.
(Probably unnecessary word of caution) do not forecast your own behavior due to risk of reduced agency.
That’s a very good point. What do you think about predicting events on which I might still have an impact? Those are some of the most important forecasts for me: I will decide whether to attempt something based on my predicted probability of success. But then my forecast might affect that probability which makes the whole thing much more complicated.
Agree! Tricky territory. I think it’s fair to take an outside view as a first cut (e.g. how many people survived Everest), then very carefully evaluate if the reference class is relevant. Yudkowsky writes about this quite a bit in places cannot recall which particular place.
Thanks. I will be on the lookout for relevant writings. I’m slowly going through Yudkowsky’s books/posts, so I’m sure I will stumble on it sooner or later.
Spreadsheet sounds “good enough” if you’re not sure you even want to commit to doing this.
That said, I’m “mildly interested” in doing this, but I don’t really have inspiration for questions I’d like to make predictions on. I’m not particularly interested in doing predictions about global events and would rather make predictions about personal events. I would like a site that lets me see other people’s personal predictions (really, just their questions they’re prediction an answer to—I don’t care about their actual answers), so that I can try to make the same predictions about my life. So for example, now that I’ve seen you’ve submitted a prediction for “Will my parents die this year?”, I don’t know what your answer is, but I can come up with my own answer to the question of whether or not *my* parents will die this year.
Yes, exactly. I dropped the idea of writing my own software because I realised that would be overcommitting too early.
You might like to search by tag “Personal” on the PredictionBook linked below by habryka: https://www.google.com/search?sitesearch=predictionbook.com&q=personal&x=0&y=0
I do have a spreadsheet where I keep track of predictions, though only tracking the prediction, my confidence, and whether it came true or false. It’s low effort and I think worth doing, but I can’t confidently say that it has improved my calibration.