[Question] How do you assess the quality /​ reliability of a scientific study?

When you look at a pa­per, what signs cause you to take it se­ri­ously? What signs cause you to dis­card the study as too poorly de­signed to be much ev­i­dence one way or the other?

I’m hop­ing to com­pile a repos­i­tory of heuris­tics on study eval­u­a­tion, and would love to hear peo­ple’s tips and tricks, or their full eval­u­a­tion-pro­cess.

I’m look­ing for things like...

  • “If the n (sam­ple size) is be­low [some thresh­old value], I usu­ally don’t pay much at­ten­tion.”

  • “I’m mostly on the look­out for big effect sizes.”

  • “I read the ab­stract, then I spend a few min­utes think­ing about how I would de­sign the ex­per­i­ment, in­clud­ing which con­founds I would have to con­trol for, and how I could do that. Then I read the meth­ods sec­tion, and see how their study de­sign com­pares to my 1-3 minute sketch. Does their de­sign seem sen­si­ble? Are they ac­count­ing for the first-or­der-ob­vi­ous con­founds?”

  • etc.