I am not sure if I am following you, but the issue isn’t sample size, the issue is whether your answer is biased or not. If your estimator is severely biased, it doesn’t matter what the sample size is. That is, if the effect is 0, but your bias is −3, then you estimate −3 poorly at 100 samples, and better at 100000 samples, but you are still estimating a number that is not 0.
You seem to think that if the effect is 0, then getting it to look like it is not 0 requires very exotic scenarios or sets of variables, but that’s not true at all. It is very easy to get bias. So easy that people even have word examples of it happening: “when I regressed the possession of an olympic gold medal on physical fitness, I got a strong effect, even adjusting for socioeconomic background, age, gender, and height—to the olympic medal replica store!”
“Causal pathways” are a hard problem. You have to be very careful, especially if your data tracks people over time. The keywords here are “mediation analysis.”
Pearl said that he does not deal with statistical issues, only identification. So perhaps he would not be the best person to judge a study (because statistical analysis itself is a difficult art, even if we get the causal part right). Perhaps folks at the Harvard, (or Hopkins, or Berkeley, or North Carolina? or Penn?) causal groups would be better.
Thanks. I don’t have technical knowledge of statistics, and may be off base. I’ll have to think about this more.
Do you disagree with my bottom line that the Dale Krueger study points (to a nontrivial degree) in the direction of having a prior that there’s no effect?
Hi,
I am not sure if I am following you, but the issue isn’t sample size, the issue is whether your answer is biased or not. If your estimator is severely biased, it doesn’t matter what the sample size is. That is, if the effect is 0, but your bias is −3, then you estimate −3 poorly at 100 samples, and better at 100000 samples, but you are still estimating a number that is not 0.
You seem to think that if the effect is 0, then getting it to look like it is not 0 requires very exotic scenarios or sets of variables, but that’s not true at all. It is very easy to get bias. So easy that people even have word examples of it happening: “when I regressed the possession of an olympic gold medal on physical fitness, I got a strong effect, even adjusting for socioeconomic background, age, gender, and height—to the olympic medal replica store!”
“Causal pathways” are a hard problem. You have to be very careful, especially if your data tracks people over time. The keywords here are “mediation analysis.”
Pearl said that he does not deal with statistical issues, only identification. So perhaps he would not be the best person to judge a study (because statistical analysis itself is a difficult art, even if we get the causal part right). Perhaps folks at the Harvard, (or Hopkins, or Berkeley, or North Carolina? or Penn?) causal groups would be better.
See my response to Eliezer, as well.
Thanks. I don’t have technical knowledge of statistics, and may be off base. I’ll have to think about this more.
Do you disagree with my bottom line that the Dale Krueger study points (to a nontrivial degree) in the direction of having a prior that there’s no effect?