I Defy the Data!

One of the great weak­nesses of Science is this mis­taken idea that if an ex­per­i­ment con­tra­dicts the dom­i­nant the­ory, we should throw out the the­ory in­stead of the ex­per­i­ment.

Ex­per­i­ments can go awry. They can con­tain de­sign flaws. They can be de­liber­ately cor­rupted. They can be un­con­sciously cor­rupted. They can be se­lec­tively re­ported. Most of all, 1 time in 20 they can be “statis­ti­cally sig­nifi­cant” by sheer co­in­ci­dence, and there are a lot of ex­per­i­ments out there.

Un­for­tu­nately, Science has this no­tion that you can never go against an hon­estly ob­tained ex­per­i­men­tal re­sult. So, when some­one ob­tains an ex­per­i­men­tal re­sult that con­tra­dicts the stan­dard model, re­searchers are faced with a dilemma for re­solv­ing their cog­ni­tive dis­so­nance: they ei­ther have to im­me­di­ately throw away the stan­dard model, or else at­tack the ex­per­i­ment—ac­cuse the re­searchers of dishon­esty, or flawed de­sign, or con­flict of in­ter­est…

Some­one once pre­sented me with a new study on the effects of in­ter­ces­sory prayer (that is, peo­ple pray­ing for pa­tients who are not told about the prayer), which showed 50% of the prayed-for pa­tients achiev­ing suc­cess at in-vitro fer­til­iza­tion, ver­sus 25% of the con­trol group. I liked this claim. It had a nice large effect size. Claims of blatant im­pos­si­ble effects are much more pleas­ant to deal with than claims of small im­pos­si­ble effects that are “statis­ti­cally sig­nifi­cant”.

So I cheer­fully said: “I defy the data.”

My origi­nal phras­ing was ac­tu­ally “I deny the data”. Nonethe­less I said it out­right, with­out apol­ogy, and with de­liber­ate in­solence. I am keep­ing my the­ory; your ex­per­i­ment is wrong.

If an ex­per­i­men­tal re­sult con­tra­dicts the Stan­dard Model, this is an im­por­tant fact. It needs to be openly ac­knowl­edged. An ex­per­i­ment that makes tra­di­tion­al­ists want to dis­card the data—or even an ex­per­i­ment that makes tra­di­tion­al­ists very skep­ti­cal of the data—should be a high pri­or­ity for repli­ca­tion. An ex­per­i­ment worth defy­ing should com­mand at­ten­tion!

But it is not so­cially ac­cept­able to say, “The hell with your ex­per­i­men­tal falsifi­ca­tion, I’m keep­ing my the­ory.” So the data has to be defied covertly—by char­ac­ter as­sas­si­na­tion of the re­searchers, by sly in­nu­en­dos, by dire hints of con­tro­versy. The data has to be dis­missed, ex­cused away, swept un­der a rug, silently into the dark, be­cause you can’t ad­mit you’re defy­ing the data. This is not a good way of fo­cus­ing at­ten­tion on an anoma­lous re­sult. This is not a good way to en­sure fund­ing for repli­ca­tion at­tempts.

It would be much bet­ter if sci­ence had a stan­dard pro­ce­dure for say­ing, “I defy the data!” It would be clearly un­der­stood that this was a bold act, and some­one else in the au­di­ence might stand up and say, “Wait a minute, is that data re­ally worth defy­ing?” If a ma­jor figure in the field said “I defy the data!“, this would be suffi­cient jus­tifi­ca­tion on grant pro­pos­als for why the re­sult ur­gently needed repli­ca­tion. Scien­tists could say, “I’m hold­ing my breath, wait­ing for repli­ca­tion,” rather than hav­ing to take sides im­me­di­ately in the char­ac­ter-as­sas­si­na­tion con­tro­versy.

Maybe you could even get the me­dia to re­port that the ex­per­i­ment has been “pub­lished but defied”. Then the repli­ca­tion, or failure to repli­cate, would be news. The repli­ca­tors could get their names in the news­pa­per, and the nega­tive re­sult could be pub­lished in a ma­jor jour­nal. If you want repli­ca­tions done, you’ll have to offer some in­cen­tive.

I would also sug­gest that when an ex­per­i­ment is defied, the repli­ca­tion must pre-de­clare a min­i­mum effect size, and at­tain sig­nifi­cance of p<0.01. In ex­treme cases where claims have been made and shot down be­fore, p<0.001.

Oh, and the prayer study? Soon enough we heard that it had been re­tracted and was prob­a­bly fraud­u­lent. But I didn’t say fraud. I didn’t spec­u­late on how the re­sults might have been ob­tained. That would have been dis­mis­sive. I just stuck my neck out, and nakedly, boldly, with­out ex­cuses, defied the data.

Ad­den­dum: I should have spel­led this out ex­plic­itly: You can defy the data on one ex­per­i­ment. You can’t defy the data on mul­ti­ple ex­per­i­ments. At that point you ei­ther have to re­lin­quish the the­ory or dis­miss the data—point to a de­sign flaw, or re­fer to an even larger body of ex­per­i­ments that failed to repli­cate the re­sult, or ac­cuse the re­searchers of a de­liber­ate hoax, et cetera. But you should not turn around and ar­gue that the the­ory and the ex­per­i­ment are ac­tu­ally com­pat­i­ble. Why didn’t you think of that be­fore you defied the data? Defy­ing the data ad­mits that the data is not com­pat­i­ble with your the­ory; it sticks your neck way out, so your head can be eas­ily chopped off.