The Decline Effect and the Scientific Method [link]

The De­cline Effect and the Scien­tific Method (ar­ti­cle @ the New Yorker)

First, as a physi­cist, I do have to point out that this ar­ti­cle con­cerns mainly softer sci­ences, e.g. psy­chol­ogy, medicine, etc.

A sum­mary of ex­pla­na­tions for this effect:

  • “The most likely ex­pla­na­tion for the de­cline is an ob­vi­ous one: re­gres­sion to the mean. As the ex­per­i­ment is re­peated, that is, an early statis­ti­cal fluke gets can­cel­led out.”

  • “Jen­nions, similarly, ar­gues that the de­cline effect is largely a product of pub­li­ca­tion bias, or the ten­dency of sci­en­tists and sci­en­tific jour­nals to pre­fer pos­i­tive data over null re­sults, which is what hap­pens when no effect is found.”

  • “Richard Palmer… sus­pects that an equally sig­nifi­cant is­sue is the se­lec­tive re­port­ing of re­sults—the data that sci­en­tists choose to doc­u­ment in the first place. … Palmer em­pha­sizes that se­lec­tive re­port­ing is not the same as sci­en­tific fraud. Rather, the prob­lem seems to be one of sub­tle omis­sions and un­con­scious mis­per­cep­tions, as re­searchers strug­gle to make sense of their re­sults.”

  • “Ac­cord­ing to Ioan­ni­dis, the main prob­lem is that too many re­searchers en­gage in what he calls “sig­nifi­cance chas­ing,” or find­ing ways to in­ter­pret the data so that it passes the statis­ti­cal test of sig­nifi­cance—the ninety-five-per-cent bound­ary in­vented by Ron­ald Fisher. … The cur­rent “ob­ses­sion” with repli­ca­bil­ity dis­tracts from the real prob­lem, which is faulty de­sign.”

Th­ese prob­lems are with the proper us­age of the sci­en­tific method, not the prin­ci­ple of the method it­self. Cer­tainly, it’s im­por­tant to ad­dress them. I think the rea­son they ap­pear so of­ten in the softer sci­ences is that biolog­i­cal en­tities are enor­mously com­plex, and so higher-level ideas that make large gen­er­al­iza­tions are more sus­cep­ti­ble to ran­dom er­ror and statis­ti­cal anoma­lies, as well as per­sonal bias, con­scious and un­con­scious.

For those who haven’t read it, take a look at Richard Feyn­man on cargo cult sci­ence if you want a good lec­ture on ex­per­i­men­tal de­sign.