“This pattern of belief is very hard to justify from a Bayesian perspective. It is just the same hypothesis in both cases. Even if, in the second case, I announce an experimental method and my intent to actually test it, I have not yet experimented and I have not yet received any observational evidence in favor of the hypothesis.”
Always consider the source.
It is the same hypothesis, but if the person telling the story is unknown to me at the outset, then the credibility of the source rises with test plans, as follows.
After the just-so story: This could be any crank making up stories to sell an idea.
After hearing of a possible test: If this person has taken the time to work out how they would test it, then my confidence is significantly increased. They’ve just set themselves apart from the pure story-tellers.
After looking at the thoroughness of their proposed tests: Confidence rises if they’ve shown a willingness to consider lots of ways that their story and testing could be wrong or misleading. Evidence of familiarity with and adherence to scientific methodology does increase credibility.
After hearing that they are committed to performing their tests: Confidence increases a little more. Not much, but a little.
Sure, these increases in confidence are small—tiny compared to the boost from actually performing even one of the tests—but they are not zero.
If I’m already familiar with the story-teller, and have some prior opinion of their experience and capability, then you’re right: hearing about the tests shouldn’t affect my confidence.
I underwent the exact change in confidence described in the above article and for pretty much those reasons, on reflection. The situation is an extremely common on in science. The only place you’re wrong is in saying that the confidence update is necessarily small.
If somebody has worked out a series of experiments to test an idea, it means they’ve thought about the idea and taken it seriously, and are willing to advertise this at the risk of some of their credibility. You’d assign it a higher prior to such an idea than an off-hand remark Not even counting the demonstration of capability it provides.
“This pattern of belief is very hard to justify from a Bayesian perspective. It is just the same hypothesis in both cases. Even if, in the second case, I announce an experimental method and my intent to actually test it, I have not yet experimented and I have not yet received any observational evidence in favor of the hypothesis.”
Always consider the source.
It is the same hypothesis, but if the person telling the story is unknown to me at the outset, then the credibility of the source rises with test plans, as follows.
After the just-so story: This could be any crank making up stories to sell an idea.
After hearing of a possible test: If this person has taken the time to work out how they would test it, then my confidence is significantly increased. They’ve just set themselves apart from the pure story-tellers.
After looking at the thoroughness of their proposed tests: Confidence rises if they’ve shown a willingness to consider lots of ways that their story and testing could be wrong or misleading. Evidence of familiarity with and adherence to scientific methodology does increase credibility.
After hearing that they are committed to performing their tests: Confidence increases a little more. Not much, but a little.
Sure, these increases in confidence are small—tiny compared to the boost from actually performing even one of the tests—but they are not zero.
If I’m already familiar with the story-teller, and have some prior opinion of their experience and capability, then you’re right: hearing about the tests shouldn’t affect my confidence.
I underwent the exact change in confidence described in the above article and for pretty much those reasons, on reflection. The situation is an extremely common on in science. The only place you’re wrong is in saying that the confidence update is necessarily small. If somebody has worked out a series of experiments to test an idea, it means they’ve thought about the idea and taken it seriously, and are willing to advertise this at the risk of some of their credibility. You’d assign it a higher prior to such an idea than an off-hand remark Not even counting the demonstration of capability it provides.