Do you think that these drugs significantly help with alcoholism (as one might posit if the drugs help significantly with willpower)? If so, I’m curious what you make of this Dynomight post arguing that so far the results don’t look promising.
I don’t think that study shows much either way: too small and underpowered to show much of anything (aside from the attrition undermining internal validity).
Dynomight’s primary criticism doesn’t hold much water because it is (un-pre-registered) reverse p-hacking. If you check enough covariates, you’ll find a failure of randomization to balance on some covariate, and you can, if you wish, tell a post hoc story about how that is actually responsible for the overall mean difference. Nevertheless, randomization works, because on average why would any particular covariate be the way in which the confounding is mediated?
I think it’s convincing that the effect, if it exists, is much smaller than the one for weight. The graph for weight is so obvious you don’t even need to do statistics.
I would not believe that unless you have done a simulation study with the small n of this study, plausible levels of measurement error (alcoholism being much harder to measure than weight or body fat), with about a dozen covariates (to correspond to the different ways to slice the patients and threshold BMI etc), and then shown that you hardly ever get a false negative like this. My experience with doing such power analysis simulation studies for other things inclines me to think that people greatly overestimate how informative such small studies are once you allow for plausible levels of measurement error and (reverse) p-hacking degrees of freedom.
Do you think that these drugs significantly help with alcoholism (as one might posit if the drugs help significantly with willpower)? If so, I’m curious what you make of this Dynomight post arguing that so far the results don’t look promising.
I don’t think that study shows much either way: too small and underpowered to show much of anything (aside from the attrition undermining internal validity).
Dynomight’s primary criticism doesn’t hold much water because it is (un-pre-registered) reverse p-hacking. If you check enough covariates, you’ll find a failure of randomization to balance on some covariate, and you can, if you wish, tell a post hoc story about how that is actually responsible for the overall mean difference. Nevertheless, randomization works, because on average why would any particular covariate be the way in which the confounding is mediated?
Just have to wait for more studies.
I think it’s convincing that the effect, if it exists, is much smaller than the one for weight. The graph for weight is so obvious you don’t even need to do statistics.
I would not believe that unless you have done a simulation study with the small n of this study, plausible levels of measurement error (alcoholism being much harder to measure than weight or body fat), with about a dozen covariates (to correspond to the different ways to slice the patients and threshold BMI etc), and then shown that you hardly ever get a false negative like this. My experience with doing such power analysis simulation studies for other things inclines me to think that people greatly overestimate how informative such small studies are once you allow for plausible levels of measurement error and (reverse) p-hacking degrees of freedom.