Thanks for the thoughtful comment. I’ll try to address these remarks in order. You state
Furthermore you only examine all cause mortality, whereas the study examines deaths from specific diseases.
They also use overall mortality (Web Table 10), which is what I was trying to reproduce and screenshotted. The significance figures aren’t really different than those for the regressions broken down by mortality cause (Web Table 15), but the rate ratios for the all cause mortality ratios are clearly smaller in the disaggregated regressions because people die from other stuff. I mostly ignored the rate ratio sizes and focused on the significances here, but agree the most powerful effects probably result from the disaggregated regressions.
Removing all 14 controls will kill your statistical power further, because the presence of things like “adequate sanitation” and “health expenditure” will definitely affect the mortality rate. … they seem plausible on priors (I’m not familiar enough to know if they’re standard), and seem to improve the predictive power of the model.
This is a fundamental misunderstanding of how controls work and what they’re supposed to do. Just yesterday Cremieux wrote a pretty good piece on this very topic. The authors include these controls with little thought to the underlying causal mechanism. Their only remarks on them at all in the supplemental material are
These controls, combined with country fixed-effects models and other isolation factors, help refine the relationship between USAID per capita and changes in mortality over time.
This isn’t necessarily true at all. Consider one of the controls: Health expenditure as a percent of GDP. Is that a confounder, influencing USAID spend levels and mortality rates simultaneously? Is it a collider, caused by increased or decreased USAID spend and mortality changes? In the former case, yes, go ahead and control for it, but if it’s the latter, it screws up the analysis. Both are plausible. The authors consider neither.
I did apparently miscount the controls. It’s unclear why their own spec on page 13 miscounts them as well.
You mention the Monte Carlo simulations aren’t comparable. This is a fair, and I really like the explanation. I didn’t really touch on that aspect of this analysis in the post, but you’ve persuaded me I’m making a cheap point.
“Also it makes no sense to say “some of the choices they make to strengthen the result end up being counterproductive”.
This was unclear, and I regret it. I meant counterproductive in the sense of complicating the analysis. I’m still not clear how they got such strong, consistent results. My suspicions is careful control selection as alluded to above.
Thanks for the thoughtful comment. I’ll try to address these remarks in order. You state
They also use overall mortality (Web Table 10), which is what I was trying to reproduce and screenshotted. The significance figures aren’t really different than those for the regressions broken down by mortality cause (Web Table 15), but the rate ratios for the all cause mortality ratios are clearly smaller in the disaggregated regressions because people die from other stuff. I mostly ignored the rate ratio sizes and focused on the significances here, but agree the most powerful effects probably result from the disaggregated regressions.
This is a fundamental misunderstanding of how controls work and what they’re supposed to do. Just yesterday Cremieux wrote a pretty good piece on this very topic. The authors include these controls with little thought to the underlying causal mechanism. Their only remarks on them at all in the supplemental material are
This isn’t necessarily true at all. Consider one of the controls: Health expenditure as a percent of GDP. Is that a confounder, influencing USAID spend levels and mortality rates simultaneously? Is it a collider, caused by increased or decreased USAID spend and mortality changes? In the former case, yes, go ahead and control for it, but if it’s the latter, it screws up the analysis. Both are plausible. The authors consider neither.
I did apparently miscount the controls. It’s unclear why their own spec on page 13 miscounts them as well.
You mention the Monte Carlo simulations aren’t comparable. This is a fair, and I really like the explanation. I didn’t really touch on that aspect of this analysis in the post, but you’ve persuaded me I’m making a cheap point.
“Also it makes no sense to say “some of the choices they make to strengthen the result end up being counterproductive”.
This was unclear, and I regret it. I meant counterproductive in the sense of complicating the analysis. I’m still not clear how they got such strong, consistent results. My suspicions is careful control selection as alluded to above.