Had to actually think about it a bit, and I think it comes down to this:
The thing that determines the strength of evidence in favor of some hypothesis vs another is “what’s the likelihood we would have seen E if H were true vs what’s the likelihood we would have seen E if H were false”
Now. experimenter B is not at all filtering based on H being true or false, but merely the properties of E.
So the fact of the experimenter presenting the evidence E to us can only (directly) potentially give us additional information on the properties of the total evidence E that was collected, rather than (directly) telling us anything about H.
But… the “filtering” rule the experimenter uses is only when to stop experimenting. In other words, once the experimenter does present data E, we know that E is all the evidence there is that he collected. In other words, this isn’t filtered evidence in the sense of the experimenter throwing away data he or she doesn’t like because once we are given E, there’s nothing more to know.
Let me clarify that: Imagine you didn’t know the difference in the second experimenter’s protocol, you had thought they were the same. Then later you learn the difference. Have you actually learned anything new? Is there any new info about E that you have that you didn’t already believe you had?
In this case, no, because unlike filtered evidence situations, the information about the experimenter’s intent has no affect on what other possible evidence there may have been that was hidden from you. The probability of you seeing this specific evidence, this specific chunk of data from experimenter A is the same as that from experimenter B, given either effectiveness or non effectiveness.
There’re other patterns of data that one would expect to be possible to see from A but not from B, and other patterns that one would expect to possibly see from B but not from A, but these specific data sets being published have probability completely independant of which experimenter was doing it, right?
Had to actually think about it a bit, and I think it comes down to this:
The thing that determines the strength of evidence in favor of some hypothesis vs another is “what’s the likelihood we would have seen E if H were true vs what’s the likelihood we would have seen E if H were false”
Now. experimenter B is not at all filtering based on H being true or false, but merely the properties of E.
So the fact of the experimenter presenting the evidence E to us can only (directly) potentially give us additional information on the properties of the total evidence E that was collected, rather than (directly) telling us anything about H.
But… the “filtering” rule the experimenter uses is only when to stop experimenting. In other words, once the experimenter does present data E, we know that E is all the evidence there is that he collected. In other words, this isn’t filtered evidence in the sense of the experimenter throwing away data he or she doesn’t like because once we are given E, there’s nothing more to know.
Let me clarify that: Imagine you didn’t know the difference in the second experimenter’s protocol, you had thought they were the same. Then later you learn the difference. Have you actually learned anything new? Is there any new info about E that you have that you didn’t already believe you had?
In this case, no, because unlike filtered evidence situations, the information about the experimenter’s intent has no affect on what other possible evidence there may have been that was hidden from you. The probability of you seeing this specific evidence, this specific chunk of data from experimenter A is the same as that from experimenter B, given either effectiveness or non effectiveness.
There’re other patterns of data that one would expect to be possible to see from A but not from B, and other patterns that one would expect to possibly see from B but not from A, but these specific data sets being published have probability completely independant of which experimenter was doing it, right?