I have no data and all I’ll talk about is my experience and my gut feelings on this topic.
The first question I ask myself is what problem am I trying to solve or what am I trying to improve? The answer for me is that I suspect that I am vastly overconfident in my predictions and that I selectively forget my worst forecasts. For example I remember being skeptical about someone after interviewing them for a job, writing as much to my supervisor and the applicant getting the job anyway. A few years later the person was doing an excellent job and I was surprised when stumbling upon my own e-mail. I had forgotten about it. On the other hand, I believe I have almost never forgotten cases where I made a good call about something.
So the first problem I want to solve is to become more humble in my predictions by making my failures more visible to myself.
The second improvement I would like to achieve is determining whether the probability numbers I attach to future events are reliable indicators or completely useless. That is calibration e.g. Brier score. I suspect these values have to be interpreted by “category” (i.e. you might have good calibration in politics but bad calibration in personal relationships) and that you only start getting useful results after one or two years and a few hundred forecasts.
I find it difficult to be sufficiently detailed and specific that allows for unambiguous resolution of the questions
Future-you is presumably not maliciously trying to deceive you, right? So the only case you need to worry about is future-you misunderstanding what you meant when you wrote the forecast.
I quite dislike how fuzzy the resolution becomes this way: did I really undertake a reasonable effort? Did I undershot or overshot it?
Do you think it very likely that present-you and future-you will have a very different perspective on what “reasonable effort” means? I would only clarify things up to the point where you trust future-you to do the right thing.
Perilous feedback loops can also creep in nonetheless: my reasonable effort for a 90% prediction might mean being more relaxed than otherwise: it’s a done deal, I might think.
I agree with these feedback loops. My perspective is that you should not strive for perfection but try to improve upon the status-quo. Even without writing down explicit predictions you will have such feedback loops. Do you think they become worse when you write the predictions down i.e. worse than when you just have this gut feeling something is a “done deal”?
You are right that making predictions and asking yourself questions that you might not have asked yourself otherwise might change your behavior. I would even say that it’s not uncommon to use predictions as a motivational tool because you don’t want to be proven wrong in front of yourself or others. The feedback loop is then purposefully built in.
One way of minimizing this might be to try to make predictions that are farther in the future and then trying to forget about them. For example make a lot of predictions so that you forget the particulars and then only look at the file a year later. This is a trade-off with updating the predictions on a regular basis with new information, which to me is more important.
Another potential solution is to ask other people (friends) to make predictions about you without telling you the details. They could give you a happiness questionnaire once every 3 months and not tell you until after resolution what they do with the data. In this case they are the ones working on their calibration. If you want to work on your own, you can make predictions about them.
Or take this simpler version:
If I switch to another job in the next 12 months, how likely is it that I’ll be more satisfied with it in the first two months than I’m now?
Hoo boy, where do we even start with this one, even though lots of people make major life decisions on exactly these kinds of hinges! What if I am just a little bit happier afterward, and it’s hard to say? Can I grade this as 60% passed (and 40% failed)?
No, I don’t think you should grade it as 60% passed. It was a yes/no question. As long as you are even a little bit happier, the answer is yes.
At evaluation, I need only concern myself with how sure I am that I’m below 9 and above 7--”or am I at only 6.8?”
When making the prediction you already knew that your judgement at resolution was going to be subjective. If you dislike that, maybe it’s not a useful prediction to make.
One way around this might be to try to make “job satisfaction” something you derive from multiple variables (e.g. take the mean value of “how nice is the office”, “how nice are the colleagues”, “how challenging are the tasks”, …). Then it won’t be obvious at resolution time how to get the result that you wanted but rather you aggregate and you roll with the result.
I am really interested in forecasting and getting better at it so I am developing Cleodora, a free and open-source tool to track such forecasts. I encourage you and other readers to have a look, leave me your thoughts and help me improve it!
I have no data and all I’ll talk about is my experience and my gut feelings on this topic.
The first question I ask myself is what problem am I trying to solve or what am I trying to improve? The answer for me is that I suspect that I am vastly overconfident in my predictions and that I selectively forget my worst forecasts. For example I remember being skeptical about someone after interviewing them for a job, writing as much to my supervisor and the applicant getting the job anyway. A few years later the person was doing an excellent job and I was surprised when stumbling upon my own e-mail. I had forgotten about it. On the other hand, I believe I have almost never forgotten cases where I made a good call about something.
So the first problem I want to solve is to become more humble in my predictions by making my failures more visible to myself.
The second improvement I would like to achieve is determining whether the probability numbers I attach to future events are reliable indicators or completely useless. That is calibration e.g. Brier score. I suspect these values have to be interpreted by “category” (i.e. you might have good calibration in politics but bad calibration in personal relationships) and that you only start getting useful results after one or two years and a few hundred forecasts.
Future-you is presumably not maliciously trying to deceive you, right? So the only case you need to worry about is future-you misunderstanding what you meant when you wrote the forecast.
Do you think it very likely that present-you and future-you will have a very different perspective on what “reasonable effort” means? I would only clarify things up to the point where you trust future-you to do the right thing.
I agree with these feedback loops. My perspective is that you should not strive for perfection but try to improve upon the status-quo. Even without writing down explicit predictions you will have such feedback loops. Do you think they become worse when you write the predictions down i.e. worse than when you just have this gut feeling something is a “done deal”?
You are right that making predictions and asking yourself questions that you might not have asked yourself otherwise might change your behavior. I would even say that it’s not uncommon to use predictions as a motivational tool because you don’t want to be proven wrong in front of yourself or others. The feedback loop is then purposefully built in.
One way of minimizing this might be to try to make predictions that are farther in the future and then trying to forget about them. For example make a lot of predictions so that you forget the particulars and then only look at the file a year later. This is a trade-off with updating the predictions on a regular basis with new information, which to me is more important.
Another potential solution is to ask other people (friends) to make predictions about you without telling you the details. They could give you a happiness questionnaire once every 3 months and not tell you until after resolution what they do with the data. In this case they are the ones working on their calibration. If you want to work on your own, you can make predictions about them.
No, I don’t think you should grade it as 60% passed. It was a yes/no question. As long as you are even a little bit happier, the answer is yes.
When making the prediction you already knew that your judgement at resolution was going to be subjective. If you dislike that, maybe it’s not a useful prediction to make.
One way around this might be to try to make “job satisfaction” something you derive from multiple variables (e.g. take the mean value of “how nice is the office”, “how nice are the colleagues”, “how challenging are the tasks”, …). Then it won’t be obvious at resolution time how to get the result that you wanted but rather you aggregate and you roll with the result.
I am really interested in forecasting and getting better at it so I am developing Cleodora, a free and open-source tool to track such forecasts. I encourage you and other readers to have a look, leave me your thoughts and help me improve it!
https://cleodora.org
https://github.com/cleodora-forecasting/cleodora