When you look at a paper, what signs cause you to take it seriously? What signs cause you to discard the study as too poorly designed to be much evidence one way or the other?
I’m hoping to compile a repository of heuristics on study evaluation, and would love to hear people’s tips and tricks, or their full evaluation-process.
I’m looking for things like...
“If the n (sample size) is below [some threshold value], I usually don’t pay much attention.”
“I’m mostly on the lookout for big effect sizes.”
“I read the abstract, then I spend a few minutes thinking about how I would design the experiment, including which confounds I would have to control for, and how I could do that. Then I read the methods section, and see how their study design compares to my 1-3 minute sketch. Does their design seem sensible? Are they accounting for the first-order-obvious confounds?”