But I was more thinking about one bias appearing stronger when a bias in another direction is eliminated. For example bias X makes people think A, bias Y makes people think B, if a person is under influence of both biases, the answer is randomly A or B. In such case, eliminating bias X leads to increase of answer B.
Biases don’t have clear ‘directions’ often. If you are overconfident on a claim P, that’s just as accurate as saying you were underconfident on claim ~P. Similarly for anchoring or priming—if you anchor on the random number generator while estimating number of African nations, whether you look “over” or “under” is going to depend on whether the RNG was spitting out 1-50 or 100-200, perhaps.
I would like to be at least 95% certain
And what does that mean? If you just want to know ‘what do smart people in general believe versus normal people’, you don’t need large samples if you can get a random selection and your questions are each independent. For example, in my recent Wikipedia experiment I removed only 100 links and 3 were reverted; when I put that into a calculator for a Bernouilli distribution, I get 99% certainty that the true reversion rate is 0-7%. So to simplify considerably, if you sampled 100 smart people and 100 dumb people and they differ by 14%, is that enough certainty for you?
Biases don’t have clear ‘directions’ often. If you are overconfident on a claim P, that’s just as accurate as saying you were underconfident on claim ~P. Similarly for anchoring or priming—if you anchor on the random number generator while estimating number of African nations, whether you look “over” or “under” is going to depend on whether the RNG was spitting out 1-50 or 100-200, perhaps.
And what does that mean? If you just want to know ‘what do smart people in general believe versus normal people’, you don’t need large samples if you can get a random selection and your questions are each independent. For example, in my recent Wikipedia experiment I removed only 100 links and 3 were reverted; when I put that into a calculator for a Bernouilli distribution, I get 99% certainty that the true reversion rate is 0-7%. So to simplify considerably, if you sampled 100 smart people and 100 dumb people and they differ by 14%, is that enough certainty for you?
I am not good at statistics, but I guess yes. Especially if those 100 people are really randomly selected, which in the given situation they were.