Possible corollary: if you want to communicate with someone who might have private information, make clear your guesses about what might not be known. Their private info is a test set for your ability to forecast. Only relevant when that guessing might actually be relevant and plausibly pro-social, and maybe this whole suggestion is a universally bad idea. But I can think of scenarios where one knows someone has private info they won’t share and you’re trying to make predictive claims, and showing you have any ability at all to guess vaguely right from public info would sure help.
You might also ruin your reputation by accidentally guessing only the secret info that is not available to them.
Imagine that there are alien spaceships in both Area 51 and Area 52. Your friend only has security clearance to know about Area 52. You only figure out the information about Area 51. After telling your friend, they will update towards you being wrong.
It may not be a very good test, in many cases. Perhaps modifying it to gauge confidence could be better?
One can imagine there are near infinite sets of things that might be true for whatever the secret knowledge is regarding. Only a subset actually is a true. What is the most probable prior may be wildly divergent from what is actually true. If you judge purely on how they confirm to your secret set that doesn’t tell you how good they are at forecasting in general, just that they happen to be wrong on that set.
If you gauge confidence, that might be better. If they are very confident about something you know to be wrong, it is unlikely that the prior probability lined up with reality. If they are only moderately confident, or believe it best explains the evidence they have but are fully aware it may be incomplete or not explain other evidence they lack, then it seems unreasonable to strongly hold a view based on them.
Possible corollary: if you want to communicate with someone who might have private information, make clear your guesses about what might not be known. Their private info is a test set for your ability to forecast. Only relevant when that guessing might actually be relevant and plausibly pro-social, and maybe this whole suggestion is a universally bad idea. But I can think of scenarios where one knows someone has private info they won’t share and you’re trying to make predictive claims, and showing you have any ability at all to guess vaguely right from public info would sure help.
You might also ruin your reputation by accidentally guessing only the secret info that is not available to them.
Imagine that there are alien spaceships in both Area 51 and Area 52. Your friend only has security clearance to know about Area 52. You only figure out the information about Area 51. After telling your friend, they will update towards you being wrong.
It may not be a very good test, in many cases. Perhaps modifying it to gauge confidence could be better?
One can imagine there are near infinite sets of things that might be true for whatever the secret knowledge is regarding. Only a subset actually is a true. What is the most probable prior may be wildly divergent from what is actually true. If you judge purely on how they confirm to your secret set that doesn’t tell you how good they are at forecasting in general, just that they happen to be wrong on that set.
If you gauge confidence, that might be better. If they are very confident about something you know to be wrong, it is unlikely that the prior probability lined up with reality. If they are only moderately confident, or believe it best explains the evidence they have but are fully aware it may be incomplete or not explain other evidence they lack, then it seems unreasonable to strongly hold a view based on them.