Are you searching for positive examples of positive bias right now, or sparing a fraction of your search on what positive bias should lead you to not see? Did you look toward light or darkness?
Your hypothesis is that positive biases are generally bad. It is thus my duty to try and disprove your idea, and see what emerges from the result.
Let’s take your example, but now the sequences are ten numbers long and the initial sequence is 2-4-6-10-12-14-16-18-20-22 (the rule is still the same). Picking a sequence at random from a given set of numbers, we have only one chance in 10! = 3628800 of coming up with one that obeys the rule. Someone following the approach you recommended would probably fist try one instance of “x,x+2,x+4...” or “x,2x,3x,...”, then start checking a few random sequences (getting “No” on each one, with near certainty). In this instance, disregarding positive bias doesn’t help (unless you do a really brutal amount of testing). This is not just an artifact of “long” sequences—had we stuck with the sequence of three numbers, but the rule was “all in ascending order, or one number above ten trillion”, then finding the right rule would be just as hard. What gives?
Even worse, suppose you started with two assumptions:
1) the sequence is x,2x,3x,4x,5x,… 10x
2) the sequence is x, x+2, x+4,… x+18
You do one or two (positive) tests of 1). They comes up “yes”. You then remember to try and disprove the hypothesis, try a hundred random sequences, coming up with “no” every time. You then accept 1).
However, had you just tried to do some positive testing of 1) and 2), you would very quickly have found out that something was wrong.
Analysis:
Testing is indeed about trying to disprove a hypothesis, and gaining confidence when you fail. But your hypothesis covers uncountably many different cases, and you can test (positively or negatively) only a very few. Unless you have some grounds to assume that this is enough (such as the uniform time and space assumptions of modern science, or some sort of nice ordering or measure on the space of hypotheses or of observations), then neither positive nor negative testing are giving you much information.
However, if you have two competing hypothesis about the world, then a little testing is enough to tell which one is correct. This is the easiest way of making progress, and should always be considered.
Verdict: Awareness of positive bias causes us to think “I may be wrong, I should check”. The correct attitude in front of these sorts of problems is the subtly different “there may be other explanations for what I see, I should find them”. The two sentiments feel similar, but lead to very different ways of tackling the problem.
Are you searching for positive examples of positive bias right now, or sparing a fraction of your search on what positive bias should lead you to not see? Did you look toward light or darkness?
Your hypothesis is that positive biases are generally bad. It is thus my duty to try and disprove your idea, and see what emerges from the result.
Let’s take your example, but now the sequences are ten numbers long and the initial sequence is 2-4-6-10-12-14-16-18-20-22 (the rule is still the same). Picking a sequence at random from a given set of numbers, we have only one chance in 10! = 3628800 of coming up with one that obeys the rule. Someone following the approach you recommended would probably fist try one instance of “x,x+2,x+4...” or “x,2x,3x,...”, then start checking a few random sequences (getting “No” on each one, with near certainty). In this instance, disregarding positive bias doesn’t help (unless you do a really brutal amount of testing). This is not just an artifact of “long” sequences—had we stuck with the sequence of three numbers, but the rule was “all in ascending order, or one number above ten trillion”, then finding the right rule would be just as hard. What gives?
Even worse, suppose you started with two assumptions: 1) the sequence is x,2x,3x,4x,5x,… 10x 2) the sequence is x, x+2, x+4,… x+18
You do one or two (positive) tests of 1). They comes up “yes”. You then remember to try and disprove the hypothesis, try a hundred random sequences, coming up with “no” every time. You then accept 1).
However, had you just tried to do some positive testing of 1) and 2), you would very quickly have found out that something was wrong.
Analysis: Testing is indeed about trying to disprove a hypothesis, and gaining confidence when you fail. But your hypothesis covers uncountably many different cases, and you can test (positively or negatively) only a very few. Unless you have some grounds to assume that this is enough (such as the uniform time and space assumptions of modern science, or some sort of nice ordering or measure on the space of hypotheses or of observations), then neither positive nor negative testing are giving you much information.
However, if you have two competing hypothesis about the world, then a little testing is enough to tell which one is correct. This is the easiest way of making progress, and should always be considered.
Verdict: Awareness of positive bias causes us to think “I may be wrong, I should check”. The correct attitude in front of these sorts of problems is the subtly different “there may be other explanations for what I see, I should find them”. The two sentiments feel similar, but lead to very different ways of tackling the problem.