Discussions of correctly calibrated cognition, e.g. tracking the predictions of pundits, successes of science, graphing one’s own accuracy with tools like PredictionBook, and so on, tend to focus on positive prediction: being right about something we did predict.
Should we also count as a calibration issue the failure to predict something that, in retrospect, should have been not only predictable but predicted? (The proverbial example is “painting yourself into a corner”.)
That issue could be captured if there were some obvious way to identify issues where predictions should be made in advance. If they fail to make predictions, they are being careless; if their predictions are incorrect, they are incorrect.
I think so, but it’s important to identify the time at which it became predictable—for example, you could only predict that you were painting yourself into a corner just prior to when you made the last brushstroke that made the strip(s) of paint covering the exit path too wide to jump over. This seems hard.
Also, you’d have to know what your utility function was going to be in the future to know that some event was even worth predicting. This seems hard, too.
Discussions of correctly calibrated cognition, e.g. tracking the predictions of pundits, successes of science, graphing one’s own accuracy with tools like PredictionBook, and so on, tend to focus on positive prediction: being right about something we did predict.
Should we also count as a calibration issue the failure to predict something that, in retrospect, should have been not only predictable but predicted? (The proverbial example is “painting yourself into a corner”.)
That issue could be captured if there were some obvious way to identify issues where predictions should be made in advance. If they fail to make predictions, they are being careless; if their predictions are incorrect, they are incorrect.
I think so, but it’s important to identify the time at which it became predictable—for example, you could only predict that you were painting yourself into a corner just prior to when you made the last brushstroke that made the strip(s) of paint covering the exit path too wide to jump over. This seems hard.
Also, you’d have to know what your utility function was going to be in the future to know that some event was even worth predicting. This seems hard, too.