Isn’t Bessel engaging in a kind of publication bias? After all, if he never stops getting the desired effect size out of a study, it never concludes, so presumably you won’t hear from it.
You may have two different treatments A and B, and both have comparable effect sizes according to the literature, but you learn that all the published studies involving B were performed by Bessel who you know engages in publication bias. The published studies for A were conducted by George who, as is widely known, doesn’t have this bias. So presumably, if you hear a study was conducted by Bessel, you should correct the reported effect size downwards when estimating the real (underlying) effect size. If you hear a study was conducted by George, you can assume no such publication bias exists, so you shouldn’t correct the reported effect size downwards.
So, if A and B have the same overall reported effect size, you should assume that the effect size of B is lower than of A.
Now assume, unbeknownst to you, Bessel didn’t actually have to withhold any studies, as the effect sizes all happened be above the desired range. Should you still correct the reported effect size downward? Answer: Yes of course, since you don’t know that this is the case. The only thing you “know” is the published effect sizes and the fact that Bessel (the person) engages in biased reporting, which is evidence that the reported effect sizes overestimate the real effect size.
This is similar to how your subjective probability that you have won the lottery is very low before you have checked the results, even if, as luck would have it, you did indeed win the lottery.
Interesting. Presumably if Bessel never got the results he wanted, he could (assuming he’s honest) continue until the negative data was enough to convince himself that he was wrong. Depending on his prior that might not happen, he could run out of money or motivation before he gave up and published a negative result. Avoiding this seems related to issues about publishing negative results and timely reporting of raw data.
With regards to the biased reporting, I’ll just mention that we would have to adjust for known bias wether we were using Bayesian or frequentist methods.
Isn’t Bessel engaging in a kind of publication bias? After all, if he never stops getting the desired effect size out of a study, it never concludes, so presumably you won’t hear from it.
You may have two different treatments A and B, and both have comparable effect sizes according to the literature, but you learn that all the published studies involving B were performed by Bessel who you know engages in publication bias. The published studies for A were conducted by George who, as is widely known, doesn’t have this bias. So presumably, if you hear a study was conducted by Bessel, you should correct the reported effect size downwards when estimating the real (underlying) effect size. If you hear a study was conducted by George, you can assume no such publication bias exists, so you shouldn’t correct the reported effect size downwards.
So, if A and B have the same overall reported effect size, you should assume that the effect size of B is lower than of A.
Now assume, unbeknownst to you, Bessel didn’t actually have to withhold any studies, as the effect sizes all happened be above the desired range. Should you still correct the reported effect size downward? Answer: Yes of course, since you don’t know that this is the case. The only thing you “know” is the published effect sizes and the fact that Bessel (the person) engages in biased reporting, which is evidence that the reported effect sizes overestimate the real effect size.
This is similar to how your subjective probability that you have won the lottery is very low before you have checked the results, even if, as luck would have it, you did indeed win the lottery.
Interesting. Presumably if Bessel never got the results he wanted, he could (assuming he’s honest) continue until the negative data was enough to convince himself that he was wrong. Depending on his prior that might not happen, he could run out of money or motivation before he gave up and published a negative result. Avoiding this seems related to issues about publishing negative results and timely reporting of raw data.
With regards to the biased reporting, I’ll just mention that we would have to adjust for known bias wether we were using Bayesian or frequentist methods.