The Air Force/Texas Coronary Atherosclerosis Prevention Study exemplifies this perfectly in a cohort of 6605 patients. Before treatment both LDL-C and apoB are good predictors of major coronary events. After one year of treatment focused on LDL-C levels, their LDL-C levels stopped being predictive for a future major coronary event (P=.162), but their apoB levels were still a strong predictor (P=.001).
Their sample size is large enough to get pretty reliable P-values, which is great! However, this means is that if there was no relationship you would be unlikely to observe such extreme results by chance, not that the variable is a good predictor.
I didn’t see any correction for multiple-hypothesis testing, and the discussion of marital status leaves me thinking that they didn’t do this at all: “[The final model] included marital status (P=0.0499) (previously, marital status narrowly missed inclusion, with P=0.052)”.
Per the original form of Goodhart’s Law “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”, so I don’t find the stated claim convincing: it’s plausible that the treatment affects observables but not outcomes.
However figures two and three are pretty compelling: the relative risk reduction is between 30-45% for almost all cohorts, measured in terms of acute major coronary events per patient-century—“as previously reported, lovastatin-mediated changes in these lipids were associated with a 37% decrease”. Seems reasonable to conclude that apoB is a better target, even though the P-values don’t seem particularly relevant and I wish they’d show scatter or kde-plots under the regression lines.
You make some good points, but thinking about the fact that researchers should correct for multiple-hypothesis testing always makes me a little sad—this almost never happens. Do you have an example where a study does this really nicely?
Also, do you have any input on the hypothesis that treating early is a worthwhile risk?
Wait, I thought LDL and HDL levels were considered vital to predicting future heart disease. Why would LDL/HDL levels during treatment be so unrelated to patient outcomes? LDL only decreased 25%. Weren’t statins used to prevent heart disease specifically because of their effects on LDL/HDL levels?
A few subtle but very important statistical nitpicks (full text of the study for reference):
Their sample size is large enough to get pretty reliable P-values, which is great! However, this means is that if there was no relationship you would be unlikely to observe such extreme results by chance, not that the variable is a good predictor.
I didn’t see any correction for multiple-hypothesis testing, and the discussion of marital status leaves me thinking that they didn’t do this at all: “[The final model] included marital status (P=0.0499) (previously, marital status narrowly missed inclusion, with P=0.052)”.
Per the original form of Goodhart’s Law “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”, so I don’t find the stated claim convincing: it’s plausible that the treatment affects observables but not outcomes.
However figures two and three are pretty compelling: the relative risk reduction is between 30-45% for almost all cohorts, measured in terms of acute major coronary events per patient-century—“as previously reported, lovastatin-mediated changes in these lipids were associated with a 37% decrease”. Seems reasonable to conclude that apoB is a better target, even though the P-values don’t seem particularly relevant and I wish they’d show scatter or kde-plots under the regression lines.
You make some good points, but thinking about the fact that researchers should correct for multiple-hypothesis testing always makes me a little sad—this almost never happens. Do you have an example where a study does this really nicely?
Also, do you have any input on the hypothesis that treating early is a worthwhile risk?
Wait, I thought LDL and HDL levels were considered vital to predicting future heart disease. Why would LDL/HDL levels during treatment be so unrelated to patient outcomes? LDL only decreased 25%. Weren’t statins used to prevent heart disease specifically because of their effects on LDL/HDL levels?