Some causal diagrams and numerical calculations clarify the argument. Call IR the score on the supposed test for intuitive vs. reflective style (for me the right answers are all intuitive), BG the score for belief in god, IQ the score on an IQ test, and G the hypothetical true intelligence. Consider the causal diagram that has an arrow from G to each of IQ, IR, and BG, and no other arrows. This is the causal diagram that controlling for intelligence is intended to rule out by finding a conditional correlation that is incompatible with it.
To really control for intelligence would require matching subjects for G value. This cannot be done, so IQ is used as a proxy, but IQ correlates only 0.8 with G. The question is then: when you control for IQ, how well are you controlling for G? How much correlation due to that possible causal diagram are you squeezing out of the relationship between IR and BG? I’ve only glanced at the paper and do not know if the authors have considered this question.
I shall suppose for the sake of example, and because this example is one that I happen to have worked out the numbers for, that all the distributions involved are multivariate Gaussian. IQ and G have a product-moment correlation of c = 0.8. By what factor does knowing someone’s IQ narrow the distribution of their G? The answer is 1/sqrt(1-c^2) = 1.667.
That doesn’t look like very much, does it? Is it enough to rule out the correlation between IR and BG as being due solely to a dependence of each on G? Well, what was the unconditional correlation between IR and BG? The paper is all F-values and NHST, but the impression I get from their Figure 1 is around 0.5 to 0.8.
So, consider a hypothetical model in which IR = G + noiseIR, BG = G + noiseBG, and IQ = G + noiseIQ, the three noises being independent. This is the model that we are trying to rule out by controlling for IQ. Choose units so that all variables have mean 0, and G has variance 1. Clearly, the conditional correlation of IR and BG given G is zero, and the experiment would find against that. But how high could the conditional correlation of IR and BG be, when what is fixed is not G but IQ?
We have IR = IQ—noiseIQ + noiseIR and BG = IQ—noiseIQ + noiseBG, so their conditional correlation given IQ is the unconditional correlation between noiseIR-noiseIQ and noiseBG-noiseIQ. Calling the variances of these noises vnIQ, vnIR, and vnBG, the correlation is 1/sqrt((1 + vnIR/vIQ)(1 + vBG/vIQ)).
Choosing our units earlier means that all of the variances vIQ, vIR, and vBG are relative to the variance of G. This tells us the value of vIQ: a correlation of c=0.8 between G + noiseIQ and G implies that vIQ = 1/c^2 − 1 = 9⁄16. We don’t know vIR or vBG, but the correlation between G + noiseIR and G + noiseBG is 1/sqrt((1 + vIR)(1 + vnBG)), and we estimated this to be around 0.5 to 0.8.
Given the ratio of vnIR to vnBG (which we do not know) this determines the correlation between IR and BG conditional on IQ. My calculations indicate that the correlation is very insensitive to the ratio vnIR/vnBG over a range of 1⁄4 to 4, so I’ll just give results where vnIR=vnBG.
For an unconditional correlation cIRBG between IR and BG ranging between 0.5 and 0.8, here are the correlations cIRBGIQ conditional on IQ:
Some causal diagrams and numerical calculations clarify the argument. Call IR the score on the supposed test for intuitive vs. reflective style (for me the right answers are all intuitive), BG the score for belief in god, IQ the score on an IQ test, and G the hypothetical true intelligence. Consider the causal diagram that has an arrow from G to each of IQ, IR, and BG, and no other arrows. This is the causal diagram that controlling for intelligence is intended to rule out by finding a conditional correlation that is incompatible with it.
To really control for intelligence would require matching subjects for G value. This cannot be done, so IQ is used as a proxy, but IQ correlates only 0.8 with G. The question is then: when you control for IQ, how well are you controlling for G? How much correlation due to that possible causal diagram are you squeezing out of the relationship between IR and BG? I’ve only glanced at the paper and do not know if the authors have considered this question.
I shall suppose for the sake of example, and because this example is one that I happen to have worked out the numbers for, that all the distributions involved are multivariate Gaussian. IQ and G have a product-moment correlation of c = 0.8. By what factor does knowing someone’s IQ narrow the distribution of their G? The answer is 1/sqrt(1-c^2) = 1.667.
That doesn’t look like very much, does it? Is it enough to rule out the correlation between IR and BG as being due solely to a dependence of each on G? Well, what was the unconditional correlation between IR and BG? The paper is all F-values and NHST, but the impression I get from their Figure 1 is around 0.5 to 0.8.
So, consider a hypothetical model in which IR = G + noiseIR, BG = G + noiseBG, and IQ = G + noiseIQ, the three noises being independent. This is the model that we are trying to rule out by controlling for IQ. Choose units so that all variables have mean 0, and G has variance 1. Clearly, the conditional correlation of IR and BG given G is zero, and the experiment would find against that. But how high could the conditional correlation of IR and BG be, when what is fixed is not G but IQ?
We have IR = IQ—noiseIQ + noiseIR and BG = IQ—noiseIQ + noiseBG, so their conditional correlation given IQ is the unconditional correlation between noiseIR-noiseIQ and noiseBG-noiseIQ. Calling the variances of these noises vnIQ, vnIR, and vnBG, the correlation is 1/sqrt((1 + vnIR/vIQ)(1 + vBG/vIQ)).
Choosing our units earlier means that all of the variances vIQ, vIR, and vBG are relative to the variance of G. This tells us the value of vIQ: a correlation of c=0.8 between G + noiseIQ and G implies that vIQ = 1/c^2 − 1 = 9⁄16. We don’t know vIR or vBG, but the correlation between G + noiseIR and G + noiseBG is 1/sqrt((1 + vIR)(1 + vnBG)), and we estimated this to be around 0.5 to 0.8.
Given the ratio of vnIR to vnBG (which we do not know) this determines the correlation between IR and BG conditional on IQ. My calculations indicate that the correlation is very insensitive to the ratio vnIR/vnBG over a range of 1⁄4 to 4, so I’ll just give results where vnIR=vnBG.
For an unconditional correlation cIRBG between IR and BG ranging between 0.5 and 0.8, here are the correlations cIRBGIQ conditional on IQ:
You would need a much higher correlation betweeen IQ and G to push cIRBGIQ close to zero: