Note that their implicit definition of “replicable” is very narrow—under their procedure, one can fail to be “replicable” simply by failing to reply to an e-mail from the authors asking for code. This is somewhat of a word play, since typically “failure to replicate” means that one is unable to get the same results as the authors while following the same procedure. Based on their discussion at the end of section 3, it appears that (at most) 9 of the 30 “failed replications” are due to actually running the code and getting different results.
Yes, there is a difference between “unable to replicate because we couldn’t even attempt to replicate” (code and/or data are missing) and “unable to replicate because we tried and the results did not match”. Either both or only the second case could be called “failure to replicate”, depends on your preferred definition.
Still, while the second case is clearly “bad science”—it’s either mistakes or fraud—the first case is “not science” because science doesn’t work by trusting the word of the researcher. A well-known example of the first case is cold fusion.
Note that their implicit definition of “replicable” is very narrow—under their procedure, one can fail to be “replicable” simply by failing to reply to an e-mail from the authors asking for code. This is somewhat of a word play, since typically “failure to replicate” means that one is unable to get the same results as the authors while following the same procedure. Based on their discussion at the end of section 3, it appears that (at most) 9 of the 30 “failed replications” are due to actually running the code and getting different results.
Yes, there is a difference between “unable to replicate because we couldn’t even attempt to replicate” (code and/or data are missing) and “unable to replicate because we tried and the results did not match”. Either both or only the second case could be called “failure to replicate”, depends on your preferred definition.
Still, while the second case is clearly “bad science”—it’s either mistakes or fraud—the first case is “not science” because science doesn’t work by trusting the word of the researcher. A well-known example of the first case is cold fusion.