We have additional information about researcher 2′s experiment. If researcher 2 didn’t look at the data before that point, then the procedures were the same, so the data should be treated the same.
If researcher 2 did check the data along the way—a reasonable enough assumption, given researcher 2′s goal—then there were other tests which all came out below 60%. There was an upswing in successes at the end, and we know it. The other experiment may well have experienced the same thing, but in experiment 2, I don’t have to look; I see it. Was there an important variable that we overlooked?
Of course, this is only due to extra information I happen to have about the latter. I haven’t bothered to check whether something similar was there for the previous, because there wasn’t anything to make it catch my attention. If I have the tools to do so, I would still like to treat them both the same—I want to see researcher 1′s results lined up by time as well. If the upswing is repeated...well, that’s funny.
Good point, there is some ordering information leaked. This is consistent with identical likelihoods for both setups—learning which permutation of arguments we’re feeding into a commutative operator (multiplication of likelihood ratios) doesn’t tell us anything about its result.
We have additional information about researcher 2′s experiment. If researcher 2 didn’t look at the data before that point, then the procedures were the same, so the data should be treated the same.
If researcher 2 did check the data along the way—a reasonable enough assumption, given researcher 2′s goal—then there were other tests which all came out below 60%. There was an upswing in successes at the end, and we know it. The other experiment may well have experienced the same thing, but in experiment 2, I don’t have to look; I see it. Was there an important variable that we overlooked?
Of course, this is only due to extra information I happen to have about the latter. I haven’t bothered to check whether something similar was there for the previous, because there wasn’t anything to make it catch my attention. If I have the tools to do so, I would still like to treat them both the same—I want to see researcher 1′s results lined up by time as well. If the upswing is repeated...well, that’s funny.
Good point, there is some ordering information leaked. This is consistent with identical likelihoods for both setups—learning which permutation of arguments we’re feeding into a commutative operator (multiplication of likelihood ratios) doesn’t tell us anything about its result.