My knowledge of statistics and research is fairly elementary. I know what the different tests / conditions are, so I can do some minor discrimination, but does anyone know anything about figuring out which things are real? (i.e. what to look for when seeing if research papers check out)
My current understanding / heuristics look like:
Try to analyze the experiment yourself.
Look for multiple studies that try to confirm the phenomenon.
It’s difficult—you are basically asking for a well-calibrated bullshit detector and those mostly get calibrated by wading through a lot of bullshit and paying careful attention to it X-)
Andrew Gelman, a stats professor at Columbia, has a sideline in snickering at bad papers, so scanning his blog might give you ideas about good approaches. See e.g. this.
Also, check out Keith Stanovich’s book How to think straight about Psychology. I am a rationality noob and find it amazing, but maybe it will help you because Keith talks about how to properly interpret scientific results in the later section of the book, AFAIK.
Gwern wrote a post about how the replication crisis means that the kind of methods that produced a lot of what’s in Cialdini’s books likely shoudn’t be trusted. The post wasn’t about specific arguments that indicated that Cialdini’s claims have been shown to be wrong. If you follow Gwern’s view than it’s unknown how wrong you are.
I’m not sure to what extent I follow that argument. A principle like scarcity is strongly reduced by many sleasy online marketers who do a lot of multivariate testing. If it wouldn’t work, they wouldn’t use it.
When it comes to detecting bullshit there’s more than just looking at the statistics. It’s very worthwhile to ask yourself whether the thesis actually makes sense based on what you know to be true based on how you live your life.
If you look at willpower-as-a-resource-that’s-glucose, it’s quite easy to see problems with that thesis. The paleo folks that try to minimize their blood glucose levels don’t tend to suffer from low willpower. The brain doesn’t consume more glucose when it’s “mentally active” than when it doesn’t.
It doesn’t really fit the experience I have with how willpower works in myself.
Willpower also has the problem that it’s like Chi. Having it in the ontology can be useful for some task but there are no willpower particles or atoms just like there are no Chi particles or atoms. Baumeister et al seem to forget that willpower is just an abstraction.
Before the replication crisis, I was talking with a psychology Ph.D. about the thesis you believed on it while I didn’t. A few years later I meet him at an LW event and he told me that I was right all along. I wasn’t right because I was better at understanding the underlying statistics but because I thought about how the finding relates to other things that are true.
It might be worthwhile to read Baumeister’s book to get a feeling of how it looks like when an eminent psychologist defends a wrong thesis in the 21st century.
Cialdini? I’m finishing “Influence” right now. I was extra skeptical during reading it since I’m freshly acquainted with the replication crisis, but googling each citation and reading through the paper is way too much work. He supports many of his claims with multiple studies and real-life anecdotes (for all that’s worth). Could you point me to the criticism of Cialdini you have read?
Cialdini is based off a comment I think I saw by Scott Alexander along the lines of “everything in Cialdini now seems to be bunk”. This is low confidence and I’m happy to revise in light of new info.
My priors on Cialdini are mainly based on how priming, which seems similar to many of his claims, doesn’t replicate well.
That is indeed very low weight. My prior is pretty shaky as-is, but that evidence shouldn’t move it much.
I thought about priming a lot while reading. Many of the results he lists are similar to priming, but priming being false doesn’t mean all results similar to it are false. One could consider a broader hypothesis encompassing all that, namely “humans can be influenced by subtle clues to their subconsciousness to a significant degree”. That’s the similarity I see with priming, both it and many of Caldini’s hypothesis follow from this premise. The priming failure would suggest it’s false, but those experiments used extremely subtle subliminal clues, as if they were designed not to work. Much of Caldini’s work affirms this broader thesis. It’s no metastudy, but the guy lists a lot of studies, all affirming it. A lot of Kahneman’s work does, too. Surely it is acceptable that humans often act on instinct (unconsciously) and that they are subconsciously influenced by their surroundings. This follows from System 1 being so prevalent in our thought.
SSC has a new open thread right now, I should ask there. Maybe Scott can clear it up.
On the Replication Crisis and examining research:
My knowledge of statistics and research is fairly elementary. I know what the different tests / conditions are, so I can do some minor discrimination, but does anyone know anything about figuring out which things are real? (i.e. what to look for when seeing if research papers check out)
My current understanding / heuristics look like:
Try to analyze the experiment yourself.
Look for multiple studies that try to confirm the phenomenon.
Look for meta-analyses of the thing.
Check on OSF if replication has happened.
Priming, power stances, willpower-as-a-resource, and everything in Cialdini is on shaky ground.
It’s difficult—you are basically asking for a well-calibrated bullshit detector and those mostly get calibrated by wading through a lot of bullshit and paying careful attention to it X-)
Andrew Gelman, a stats professor at Columbia, has a sideline in snickering at bad papers, so scanning his blog might give you ideas about good approaches. See e.g. this.
Thanks! I’ve bumped into Gelman’s stuff a few times. I’ll check it out.
I Ankified Influence—Cialdini.
How wrong am I now?
Also, check out Keith Stanovich’s book How to think straight about Psychology. I am a rationality noob and find it amazing, but maybe it will help you because Keith talks about how to properly interpret scientific results in the later section of the book, AFAIK.
Gwern wrote a post about how the replication crisis means that the kind of methods that produced a lot of what’s in Cialdini’s books likely shoudn’t be trusted. The post wasn’t about specific arguments that indicated that Cialdini’s claims have been shown to be wrong. If you follow Gwern’s view than it’s unknown how wrong you are.
I’m not sure to what extent I follow that argument. A principle like scarcity is strongly reduced by many sleasy online marketers who do a lot of multivariate testing. If it wouldn’t work, they wouldn’t use it.
When it comes to detecting bullshit there’s more than just looking at the statistics. It’s very worthwhile to ask yourself whether the thesis actually makes sense based on what you know to be true based on how you live your life.
If you look at willpower-as-a-resource-that’s-glucose, it’s quite easy to see problems with that thesis. The paleo folks that try to minimize their blood glucose levels don’t tend to suffer from low willpower. The brain doesn’t consume more glucose when it’s “mentally active” than when it doesn’t. It doesn’t really fit the experience I have with how willpower works in myself.
Willpower also has the problem that it’s like Chi. Having it in the ontology can be useful for some task but there are no willpower particles or atoms just like there are no Chi particles or atoms. Baumeister et al seem to forget that willpower is just an abstraction.
Before the replication crisis, I was talking with a psychology Ph.D. about the thesis you believed on it while I didn’t. A few years later I meet him at an LW event and he told me that I was right all along. I wasn’t right because I was better at understanding the underlying statistics but because I thought about how the finding relates to other things that are true.
It might be worthwhile to read Baumeister’s book to get a feeling of how it looks like when an eminent psychologist defends a wrong thesis in the 21st century.
I find this hard to believe.
Cialdini? I’m finishing “Influence” right now. I was extra skeptical during reading it since I’m freshly acquainted with the replication crisis, but googling each citation and reading through the paper is way too much work. He supports many of his claims with multiple studies and real-life anecdotes (for all that’s worth). Could you point me to the criticism of Cialdini you have read?
Cialdini is based off a comment I think I saw by Scott Alexander along the lines of “everything in Cialdini now seems to be bunk”. This is low confidence and I’m happy to revise in light of new info.
My priors on Cialdini are mainly based on how priming, which seems similar to many of his claims, doesn’t replicate well.
That is indeed very low weight. My prior is pretty shaky as-is, but that evidence shouldn’t move it much.
I thought about priming a lot while reading. Many of the results he lists are similar to priming, but priming being false doesn’t mean all results similar to it are false. One could consider a broader hypothesis encompassing all that, namely “humans can be influenced by subtle clues to their subconsciousness to a significant degree”. That’s the similarity I see with priming, both it and many of Caldini’s hypothesis follow from this premise. The priming failure would suggest it’s false, but those experiments used extremely subtle subliminal clues, as if they were designed not to work. Much of Caldini’s work affirms this broader thesis. It’s no metastudy, but the guy lists a lot of studies, all affirming it. A lot of Kahneman’s work does, too. Surely it is acceptable that humans often act on instinct (unconsciously) and that they are subconsciously influenced by their surroundings. This follows from System 1 being so prevalent in our thought.
SSC has a new open thread right now, I should ask there. Maybe Scott can clear it up.