My interpretation is that: as an oracle varies its prediction, it both picks up points from becoming more accurate and from making the outcome more predictable. This means the proper scoring rules, tuned to only one pressure, will result in a dishonest equilibrium instead of an honest one.
(Incidentally, this seems like a pretty sensible explanation for why humans are systematically overconfident; if confidence increases success, then the point when the benefits of increased success match the costs of decreased accuracy will be more extreme than the truth, in both directions.)
We say that a prediction is a self-fulfilling prophecy or a fixed point if it is equal to the model’s beliefs about the world, after the model makes that prediction.
I think a broad definition works for ‘fixed points’, but I think for ‘self-fulfilling prophecy’ I would also expect there to be a counterfactual element—if I say the sun will rise tomorrow and it rises tomorrow, this isn’t a self-fulfilling prophecy because the outcome isn’t reliant on expectations about the outcome.
Your interpretation sounds right to me. I would add that our result implies that it is impossible to incentivize honest reports in our setting. If you want to incentivize honest reports when f is constant, then you have to use a strictly proper scoring rule (this is just the definition of “strictly proper”). But we show for any strictly proper scoring rule that there is a function f such that a dishonest prediction is optimal.
Proposition 13 shows that it is possible to “tune” scoring rules to make optimal predictions very close to honest ones (at least in L1-distance).
I think for ‘self-fulfilling prophecy’ I would also expect there to be a counterfactual element—if I say the sun will rise tomorrow and it rises tomorrow, this isn’t a self-fulfilling prophecy because the outcome isn’t reliant on expectations about the outcome.
Yes, that is fair. To be faithful to the common usage of the term, one should maybe require at least two possible fixed points (or points that are somehow close to fixed points). The case with a unique fixed point is probably also safer, and worries about “self-fulfilling prophecies” don’t apply to the same degree.
But we show for any strictly proper scoring rule that there is a function f such that a dishonest prediction is optimal.
Agreed for proper scoring rules, but I’d be a little surprised if it’s not possible to make a skill-free scoring rule, and then get a honest prediction result for that. [This runs into other issues—if the scoring rule is skill-free, where does the skill come from?--but I think this can be solved by having oracle-mode and observation-mode, and being able to do honest oracle-mode at all would be nice.]
Sure, points from a scoring rule come both from ‘skill’ (whether or not you’re accurate in your estimates) and ‘calibration’ (whether your estimates line up with the underlying propensity).
Rather than generating the picture I’m thinking of (sorry, up to something else and so just writing a quick comment), I’ll describe it: watch this animation, and see the implied maximum expected score as a function of p (the forecaster’s true belief). For all of the scoring rules, it’s a convex function with maxima at 0 and 1. (You can get 1 point on average with a linear rule if p=0, and only 0.5 points on average if p=0.5; for a log rule, it’s 0 points and −0.7 points.)
But could you come up with a scoring rule where the maximum expected score as a function of p is flat? If true, there’s no longer an incentive to have extreme probabilities. But that incentive was doing useful work before, and so this seems likely to break something else—it’s probably no longer the case that you’re incentivized to say your true belief—or require something like batch statistics (since I think you might be able to get something like this by scoring not individual predictions but sets of them, sorted by p or by whether they were true or false). [This can be done in some contexts with markets, where your reward depends on how close the market was to the truth before, but I think it probably doesn’t help here because we’re worried about the oracle’s ability to affect the underlying reality, which is also an issue with prediction markets!]
To be clear, I’m not at all confident this is possible or sensible—it seems likely to me that an adversarial argument goes thru where as oracle I always benefit from knowing which statements are true and which statements are false (even if I then lie about my beliefs to get a good calibration curve or w/e)--but that’s not an argument about the scale of the distortions that are possible.
Great post!
My interpretation is that: as an oracle varies its prediction, it both picks up points from becoming more accurate and from making the outcome more predictable. This means the proper scoring rules, tuned to only one pressure, will result in a dishonest equilibrium instead of an honest one.
(Incidentally, this seems like a pretty sensible explanation for why humans are systematically overconfident; if confidence increases success, then the point when the benefits of increased success match the costs of decreased accuracy will be more extreme than the truth, in both directions.)
I think a broad definition works for ‘fixed points’, but I think for ‘self-fulfilling prophecy’ I would also expect there to be a counterfactual element—if I say the sun will rise tomorrow and it rises tomorrow, this isn’t a self-fulfilling prophecy because the outcome isn’t reliant on expectations about the outcome.
Thanks for your comment!
Your interpretation sounds right to me. I would add that our result implies that it is impossible to incentivize honest reports in our setting. If you want to incentivize honest reports when f is constant, then you have to use a strictly proper scoring rule (this is just the definition of “strictly proper”). But we show for any strictly proper scoring rule that there is a function f such that a dishonest prediction is optimal.
Proposition 13 shows that it is possible to “tune” scoring rules to make optimal predictions very close to honest ones (at least in L1-distance).
Yes, that is fair. To be faithful to the common usage of the term, one should maybe require at least two possible fixed points (or points that are somehow close to fixed points). The case with a unique fixed point is probably also safer, and worries about “self-fulfilling prophecies” don’t apply to the same degree.
Agreed for proper scoring rules, but I’d be a little surprised if it’s not possible to make a skill-free scoring rule, and then get a honest prediction result for that. [This runs into other issues—if the scoring rule is skill-free, where does the skill come from?--but I think this can be solved by having oracle-mode and observation-mode, and being able to do honest oracle-mode at all would be nice.]
I’m not sure I understand what you mean by a skill-free scoring rule. Can you elaborate what you have in mind?
Sure, points from a scoring rule come both from ‘skill’ (whether or not you’re accurate in your estimates) and ‘calibration’ (whether your estimates line up with the underlying propensity).
Rather than generating the picture I’m thinking of (sorry, up to something else and so just writing a quick comment), I’ll describe it: watch this animation, and see the implied maximum expected score as a function of p (the forecaster’s true belief). For all of the scoring rules, it’s a convex function with maxima at 0 and 1. (You can get 1 point on average with a linear rule if p=0, and only 0.5 points on average if p=0.5; for a log rule, it’s 0 points and −0.7 points.)
But could you come up with a scoring rule where the maximum expected score as a function of p is flat? If true, there’s no longer an incentive to have extreme probabilities. But that incentive was doing useful work before, and so this seems likely to break something else—it’s probably no longer the case that you’re incentivized to say your true belief—or require something like batch statistics (since I think you might be able to get something like this by scoring not individual predictions but sets of them, sorted by p or by whether they were true or false). [This can be done in some contexts with markets, where your reward depends on how close the market was to the truth before, but I think it probably doesn’t help here because we’re worried about the oracle’s ability to affect the underlying reality, which is also an issue with prediction markets!]
To be clear, I’m not at all confident this is possible or sensible—it seems likely to me that an adversarial argument goes thru where as oracle I always benefit from knowing which statements are true and which statements are false (even if I then lie about my beliefs to get a good calibration curve or w/e)--but that’s not an argument about the scale of the distortions that are possible.