[Question] Personal predictions for decisions: seeking insights

Do some of you keep a record of personal predictions?

I.e. either predictions of personal events, such as

likelihood of getting at least a 10% raise in the next 12 months

or events conditional on decisions, actions, such as

likelihood of getting at least 10% higher income in the next 12 months given that I search for other jobs instead of not

If yes, how useful do you find this? Have you validated it in some way? Do you keep yourself honest this way by looking at your track record? Are you trying to track and improve your calibration and priors this way?

I’m quite interested in this. However, after a naive try in a spreadsheet file, which was easy to set up, I find myself at some stumbling blocks. I find it difficult to be sufficiently detailed and specific that allows for unambiguous resolution of the questions (after which brier scores can easily be attained), and making them sufficiently conditional so as to avoid interfering feedback loops, while not having to spend too much time writing the scenarios themselves.

Let’s see the above sentences as an example:

likelihood of getting at least 10% higher income in the next 12 months given that I search for other jobs instead of not

Having such a prediction sounds very useful at a first glance to me, it could help me assess if I should look for a different job! Whether it’s worth it in expectation, and how much effort might be worth expanding on it.

I.e. if the answer is 90%, that’s a good signal that I might need to take a shot at it.

If the answer is 1%, that’s a good signal that maybe I shouldn’t bother, or very little.

But let’s look closer: at second glance it seems woefully underspecified:

Just how much effort does this mean? Literally spending a single second on a job board? How can I avoid gaming myself? Let’s try to specify this better:

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings

This is better, but again what do we mean here? Should I just apply to the first 10 that I see?

I can think of a few general ways to try to solve this, to try to bring in line the actions and the predictions:

I could implicitly or explicitly append “reasonable effort” or something similar to these questions, i.e.

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings with reasonable effort

Ok, so I should be somewhat discerning in this case, but I quite dislike how fuzzy the resolution becomes this way: did I really undertake a reasonable effort? Did I undershot or overshot it?

Perilous feedback loops can also creep in nonetheless: my reasonable effort for a 90% prediction might mean being more relaxed than otherwise: it’s a done deal, I might think. Having it lower might motivate me more, thinking that I have to improve my chances, but having it too low might de-motivate me and reasonable effort could in fact be very low effort in this case. All too fuzzy!

The other way would be to instead change what I am estimating, and already assume that I’ll minimize effort in advance, and if the question is written like:

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings

Then I should estimate literally just sending my CV to the first 10 companies that I can come across, meaning my prediction should not be much any higher than not doing anything, because notice how sneaky this assumption is: I did not specify that I’ll have to subject myself to being interviewed too, so I must assume I’ll ignore all 10 even if all of them wants to interview me. Or something even more unreasonable like sending an application letter with no CV in sight.

Now we venture into more of an unknown territory: could I grade the resolutions somehow? Traditional forecasting wisdom as I understand would say “Never!”: a resolution either fully happened or fully not, or if it’s in any way ambiguous, then it fully doesn’t count, as if never made in the first place. Not ideal. But what if I could try to estimate reasonable job hunting effort, and then later grade myself of how much I actually did? Not sure how the math could work out here.

Or take this simpler version:

If I switch to another job in the next 12 months, how likely is it that I’ll be more satisfied with it in the first two months than I’m now?

Hoo boy, where do we even start with this one, even though lots of people make major life decisions on exactly these kinds of hinges! What if I am just a little bit happier afterward, and it’s hard to say? Can I grade this as 60% passed (and 40% failed)?

There could be another kind of prediction to the rescue: estimating a value instead of a probability:

If I switch to another job in the next 12 months what’s my expected satisfaction with it in the first 2 months on a scale of 1-10?

This is better. I can say 8, with 7-9 being the 95% confidence interval! This can be calculated with! At evaluation, I need only concern myself with how sure I am that I’m below 9 and above 7--”or am I at only 6.8?”

As you can see, *gestures at own confusion*, I’m a bit lost with all this. I’m familiar with superforecasting as a concept, and prediction markets, but both of these seem to be for multi-person bets that have at least some wider appeal and relevance. Calibration as a concept still seems applicable, but it’s usually in the service of the above.

But as I tried to indicate, some questions can be very important for a single person and not to others, so what can be done in this case, if one wants to improve how rational they are? Can this game be played solo? I tried searching Google, using metaphor.systems, asking LLMs, and I haven’t found satisfactory answers, so I turn to you.

And maybe the answer to all this is that one has to bite the bullet, and really go into the nitty gritty when writing questions, drill down into acceptance criteria, and then resolution is straightforward, and prediction can take all that into account.

I could try to specify further this way:

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings, go through their interview process, and if offered accept at least one such offer that is a net-improvement all things considered in expectation to my current job

(I tried to be more specific above, but I might just have exchanged a problem to a harder problem: will I be able to aggregate and assess “net-improvement all things considered”? If the difference is large, sure, but otherwise unclear.)

Or maybe for some people the “reasonable effort” or similar condition works, but I’ll be curious how you don’t fall into all kinds of problems here: fuzziness of resolutions and perilous negative and positive feedback loops, e.g. potentially divergent predictions.

And maybe the moral of this story is that this tool should only be reached to if a question is important enough that one already knows that spending 5, 10, 30 or even more minutes in really pinning it down is worth it in expectation, otherwise it becomes an exercise in futility as soon as a resolution needs to be chosen.

I’d like to hear from all of you who have experience with this or have relevant insights, or can point me towards those who do. Also feel free to recommend me any other fora where I could post this and it may be more relevant; e.g. I looked for a general conversational or Q&A forum for Metaculus but I did not find one.