But putting those aside, I find it curious that the results on psychological biases that is given so much weight on this blog are arrived at and affirmed by means of error statistical methodology.
Speaking as one of the LWers who has spent a fair bit of time reading up on both the heuristics & biases literature and also the problems & misuse of NHST (although I certainly couldn’t compare to your general statistical expertise), my position is basically that there’s no available literature which have examined the H&B topic with a superior methodology (so there’s no alternative we could use) and that on the whole H&B has found real effects despite the serious weaknesses in the methodology—for example, of the Reproducibility Project’s 13 targets, the ones which failed to replicate were priming effects and not the tested H&B effects (eg. sunk costs, anchoring, framing). The problems are not so bad as to drain the H&B results of all validity, just some.
So while the H&B research program is no doubt undermined and hampered by the statistical tools and practices of the researchers involved, there seem little reason to think that the most-discussed biases are pure statistical mirages; and so they are entirely relevant to our discussions here.
(From my perspective, the real question about the utility of the H&B literature to our practical discussions here on LW is not whether they exist in the lab settings they are studied in—it’s clear that they are not artifacts of p-value hacking or anything like that—but whether they operate in the real world to a meaningful extent and shape opinions & actions on a wide scale and on the topics we care about. This is, unfortunately, something which is very difficult to study no matter what methodology one might choose to use, and for this concern, criticizing the use of error statistical methodology is largely irrelevant.)
Speaking as one of the LWers who has spent a fair bit of time reading up on both the heuristics & biases literature and also the problems & misuse of NHST (although I certainly couldn’t compare to your general statistical expertise), my position is basically that there’s no available literature which have examined the H&B topic with a superior methodology (so there’s no alternative we could use) and that on the whole H&B has found real effects despite the serious weaknesses in the methodology—for example, of the Reproducibility Project’s 13 targets, the ones which failed to replicate were priming effects and not the tested H&B effects (eg. sunk costs, anchoring, framing). The problems are not so bad as to drain the H&B results of all validity, just some.
So while the H&B research program is no doubt undermined and hampered by the statistical tools and practices of the researchers involved, there seem little reason to think that the most-discussed biases are pure statistical mirages; and so they are entirely relevant to our discussions here.
(From my perspective, the real question about the utility of the H&B literature to our practical discussions here on LW is not whether they exist in the lab settings they are studied in—it’s clear that they are not artifacts of p-value hacking or anything like that—but whether they operate in the real world to a meaningful extent and shape opinions & actions on a wide scale and on the topics we care about. This is, unfortunately, something which is very difficult to study no matter what methodology one might choose to use, and for this concern, criticizing the use of error statistical methodology is largely irrelevant.)