Unfortunately a lot of academic papers report false or much exaggerated conclusions (https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-the-syllabus/). I have found from 30 years of reading them that there is no substitute for a) going through the methods closely and with a suspicious mind b) looking at the declared affiliations and sponsorhip and even looking outside the paper for affiliations and sponsorship. because sponsorship hugely skews results c) Closely looking at the statistics, which in turn requires a good understanding of statistics. If you do not or cannot do these things you are just reading headlines in my bitter experience.
For someone like who feels a little inadequate analyzing things like experimental design / statistical analysis, I still largely agree with this.
My shallow experience with psychology papers is that abstracts still often try to be grander than what the paper’s actually about, and looking into exactly what they did is important. Some obvious things like looking at methodology (e.g. “How did they attempt to measure the thing they claim to have measured?”) and sample sizes (“Was this done with 16 people or 1600?”) can still give you a better idea.
EX: I recall reading a study about budding altruism/charitability in young children that felt very contrived, involving a story around magic boxes, marshallows, and a teddy bear. In this case, I might be skeptical that the results they tried to generalize from were actually there.
Unfortunately a lot of academic papers report false or much exaggerated conclusions (https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-the-syllabus/). I have found from 30 years of reading them that there is no substitute for a) going through the methods closely and with a suspicious mind b) looking at the declared affiliations and sponsorhip and even looking outside the paper for affiliations and sponsorship. because sponsorship hugely skews results c) Closely looking at the statistics, which in turn requires a good understanding of statistics. If you do not or cannot do these things you are just reading headlines in my bitter experience.
For someone like who feels a little inadequate analyzing things like experimental design / statistical analysis, I still largely agree with this.
My shallow experience with psychology papers is that abstracts still often try to be grander than what the paper’s actually about, and looking into exactly what they did is important. Some obvious things like looking at methodology (e.g. “How did they attempt to measure the thing they claim to have measured?”) and sample sizes (“Was this done with 16 people or 1600?”) can still give you a better idea.
EX: I recall reading a study about budding altruism/charitability in young children that felt very contrived, involving a story around magic boxes, marshallows, and a teddy bear. In this case, I might be skeptical that the results they tried to generalize from were actually there.