tl;dr: I was excited by this post, but so far I find reading the cited literature uncompelling :( Can you point us to a study we can read where the authors reported enough of their data and procedure that we can all tell that their conclusion was justified?
I do trust you, Yvain, and I know you know stats, and I even agree with the conclusion of the post—that people are imperfect introspectors—but I’m discouraged to continue searching through the literature myself at the moment because the first two articles you cited just weren’t clear enough on what they were doing and measuring for me to tell if their conclusions were justified, other than by intuition (which I already share).
For example, none of your summaries says whether the fraction of people noticing the experimenters’ effect on their behavior was enough to explain the difference between the two experiment groups, and this seems representative of the 1977 review article you cited as your main source as well.
I looked in more detail at your first example, the electric shocks experiment (Nisbett & Schachter, 1966), on which you report
… people who took the pill tolerated four times as strong a shock as controls … Only three of twelve subjects made a connection between the pill and their shock tolerance …
I was wondering, did the experimenters merely observe
(1) a “Statistically Significant” difference between PILL-GROUP and CONTROL-GROUP? And then say “Only 3 of 12 people in the pill group managed to detect the effect of the placebo on themselves?”
Because that’s not a surprise, given the null hypothesis that people are good introspectors… maybe just those three people were affected, and that caused the significant difference between the groups! And jumping to conclusions from (1) is a kind of mistake I’ve seen before from authors assuming (if not in their minds, at least in their statistical formulae) that an effect is uniform across people, when it clearly probably isn’t.
Or, did the experimenters observe that
(2) believing that only those three subjects were actually affected by (their knowledge of) the pill was not enough to explain the difference between the groups?
To see what the study really found, after many server issues with the journal website I tracked down the original 1966 article, which I’ve made available here. The paper doesn’t mention anything about people’s assessments of whether being (told they were) given a pill may have affected their pain tolerance.
Wondering why you wrote that, I went to the 1977 survey article you read, which I’ve made available as a searchable pdf here. There they say, at the bottom left of page 237, that their conclusion about the electric shocks vs pills was based on “additional unpublished data, collected from … experiments by Nisbett and Schachter (1966)”. But their description of that was almost as terse as your summary, and in particular, included no statistical reasoning.
Like I said, I do intuitively agree with the conclusion that people are imperfect introspectors, but I worry that the authors and reviewers of this article may have been sloppy in finding clear, quantitative evidence for this perspective, perhaps by being already too convinced of it...
tl;dr: I was excited by this post, but so far I find reading the cited literature uncompelling :( Can you point us to a study we can read where the authors reported enough of their data and procedure that we can all tell that their conclusion was justified?
I do trust you, Yvain, and I know you know stats, and I even agree with the conclusion of the post—that people are imperfect introspectors—but I’m discouraged to continue searching through the literature myself at the moment because the first two articles you cited just weren’t clear enough on what they were doing and measuring for me to tell if their conclusions were justified, other than by intuition (which I already share).
For example, none of your summaries says whether the fraction of people noticing the experimenters’ effect on their behavior was enough to explain the difference between the two experiment groups, and this seems representative of the 1977 review article you cited as your main source as well.
I looked in more detail at your first example, the electric shocks experiment (Nisbett & Schachter, 1966), on which you report
I was wondering, did the experimenters merely observe
(1) a “Statistically Significant” difference between PILL-GROUP and CONTROL-GROUP? And then say “Only 3 of 12 people in the pill group managed to detect the effect of the placebo on themselves?”
Because that’s not a surprise, given the null hypothesis that people are good introspectors… maybe just those three people were affected, and that caused the significant difference between the groups! And jumping to conclusions from (1) is a kind of mistake I’ve seen before from authors assuming (if not in their minds, at least in their statistical formulae) that an effect is uniform across people, when it clearly probably isn’t.
Or, did the experimenters observe that
(2) believing that only those three subjects were actually affected by (their knowledge of) the pill was not enough to explain the difference between the groups?
To see what the study really found, after many server issues with the journal website I tracked down the original 1966 article, which I’ve made available here. The paper doesn’t mention anything about people’s assessments of whether being (told they were) given a pill may have affected their pain tolerance.
Wondering why you wrote that, I went to the 1977 survey article you read, which I’ve made available as a searchable pdf here. There they say, at the bottom left of page 237, that their conclusion about the electric shocks vs pills was based on “additional unpublished data, collected from … experiments by Nisbett and Schachter (1966)”. But their description of that was almost as terse as your summary, and in particular, included no statistical reasoning.
Like I said, I do intuitively agree with the conclusion that people are imperfect introspectors, but I worry that the authors and reviewers of this article may have been sloppy in finding clear, quantitative evidence for this perspective, perhaps by being already too convinced of it...