I don’t feel sufficiently comfortable with statistics to tear apart the given example. I do have a different example with which to refute the point that the evidential impact of a fixed set of data should be independent of the researchers prior private thoughts.
Suppose I have two researchers, both looking at the correlations between acne and colored jelly beans. Alfred does twenty tests each with X subjects. Each test will feed subjects jelly beans of a single color for a week and then look at incidences of acne. Boris theorizes that green jelly beans are correlated with acne. Boris then does twenty test each with X subjects, identical to Alfred’s test.
Alfred and Boris each use the exact same experimental procedure and each get the exact same results, finding higher rates of acne in subjects fed green jelly beans than in subjects fed other colored jelly beans. Boris’ experiment is stronger evidence for a link between green jelly beans and acne than Alfred’s experiment. Why? Because coincidences happen all the time.
Boris was looking for a correlation between green jelly beans and acne and the odds that Boris would find a correlation between green jelly beans and acne (by chance alone) was very low. Alfred was looking for a correlation but he wasn’t specific about what correlation he was looking for. By chance alone, he was just as likely to find a correlation between blue jelly beans, or red jelly beans, or any of the 17 other colors in the experiment. The fact that this experiment happened to show higher rates with green jelly beans isn’t worth very much evidence. Now if Alfred were to use this experiment to form a hypothesis that green jelly beans were correlated with acne and perform another experiment which ALSO showed a relationship between green jelly beans and acne, THEN he would have much stronger evidence.
Having written this out, I’ll offer a simpler example. Charlie and David each have a six sided die. They both think his die is weighted. Charlie says “I think this die will land ‘6’ when I roll it”. He rolls the die and it lands ‘6’. David just rolls the die and it lands ‘6’. They both now believe that his die will always land ‘6’. Exact same evidence. Charlie is more justified in his belief that the die is weighted to land ‘6’ because if his die was not weighted, there would only be a 1⁄6 chance that he would have rolled a ‘6’. If David’s die was not weighted, there would be a 6⁄6 chance that his die would roll to SOMETHING and generated his belief.
When evaluating the rational evidential impact of the results of an experiment, it is imperative that you take into account what it is that is being tested, and that is something that only exists in the experimenters private thoughts (or notebook or whatnot).
I don’t feel sufficiently comfortable with statistics to tear apart the given example. I do have a different example with which to refute the point that the evidential impact of a fixed set of data should be independent of the researchers prior private thoughts.
Suppose I have two researchers, both looking at the correlations between acne and colored jelly beans. Alfred does twenty tests each with X subjects. Each test will feed subjects jelly beans of a single color for a week and then look at incidences of acne. Boris theorizes that green jelly beans are correlated with acne. Boris then does twenty test each with X subjects, identical to Alfred’s test.
Alfred and Boris each use the exact same experimental procedure and each get the exact same results, finding higher rates of acne in subjects fed green jelly beans than in subjects fed other colored jelly beans. Boris’ experiment is stronger evidence for a link between green jelly beans and acne than Alfred’s experiment. Why? Because coincidences happen all the time.
Boris was looking for a correlation between green jelly beans and acne and the odds that Boris would find a correlation between green jelly beans and acne (by chance alone) was very low. Alfred was looking for a correlation but he wasn’t specific about what correlation he was looking for. By chance alone, he was just as likely to find a correlation between blue jelly beans, or red jelly beans, or any of the 17 other colors in the experiment. The fact that this experiment happened to show higher rates with green jelly beans isn’t worth very much evidence. Now if Alfred were to use this experiment to form a hypothesis that green jelly beans were correlated with acne and perform another experiment which ALSO showed a relationship between green jelly beans and acne, THEN he would have much stronger evidence.
Jelly beans inspired by XKCD: http://xkcd.com/882/
Having written this out, I’ll offer a simpler example. Charlie and David each have a six sided die. They both think his die is weighted. Charlie says “I think this die will land ‘6’ when I roll it”. He rolls the die and it lands ‘6’. David just rolls the die and it lands ‘6’. They both now believe that his die will always land ‘6’. Exact same evidence. Charlie is more justified in his belief that the die is weighted to land ‘6’ because if his die was not weighted, there would only be a 1⁄6 chance that he would have rolled a ‘6’. If David’s die was not weighted, there would be a 6⁄6 chance that his die would roll to SOMETHING and generated his belief.
When evaluating the rational evidential impact of the results of an experiment, it is imperative that you take into account what it is that is being tested, and that is something that only exists in the experimenters private thoughts (or notebook or whatnot).
If you have twenty minutes, I would recommend this portion of a Radiolab broadcast about coincidences: http://www.radiolab.org/2009/jun/15/a-very-lucky-wind/