I do want to point out that, while I agree with your general points, I think that unless the proponents put numerical estimates up beforehand, it’s not quite fair to assume they meant “it will be statistically significant in a sample size of N at least 95% of the time.” Even if they said that, unless they explicitly calculated N, they probably underestimated it by at least one order of magnitude. (Professional researchers in social science make this mistake very frequently, and even when they avoid it, they can only very rarely find funding to actually collect N samples.)
I haven’t looked into this study in depth, so semi-related anecdote time: there was recently a study of calorie restriction in monkeys which had ~70 monkeys. The confidence interval for the hazard ratio included 1 (no effect), and so they concluded no statistically significant benefit to CR on mortality, though they could declare statistically significant benefit on a few varieties of mortality and several health proxies.
I ran the numbers to determine the power; turns out that they couldn’t have reliably noticed the effects of smoking (hazard ratio ~2) on longevity with a study of ~70 monkeys, and while I haven’t seen many quoted estimates of the hazard ratio of eating normally compared to CR, I don’t think there are many people that put them higher than 2.
When you don’t have the power to reliably conclude that all-cause mortality decreased, you can eke out some extra information by looking at the signs of all the proxies you measured. If insurance does nothing, we should expect to see the effect estimates scattered around 0. If insurance has a positive effect, we should expect to see more effect estimates above 0 than below 0, even though most will include 0 in their CI. (Suppose they measure 30 mortality proxies, and all of them show a positive effect, though the univariate CI includes 0 for all of them. If the ground truth was no effect on mortality proxies, that’s a very unlikely result to see; if the ground truth was a positive effect on mortality proxies, that’s a likely result to see.)
I ran the numbers to determine the power; turns out that they couldn’t have reliably noticed the effects of smoking (hazard ratio ~2) on longevity with a study of ~70 monkeys, and while I haven’t seen many quoted estimates of the hazard ratio of eating normally compared to CR, I don’t think there are many people that put them higher than 2.
If I remember correctly, I noticed an effect that did give a p of slightly less than .05 was a hazard ratio of 3, which made me think of running that test, and then I think spower was the r function that I used to figure out what p they could get for a hazard ratio of 2 and 35 experimentals and 35 controls (or whatever the actual split was- I think it was slightly different?).
So you were using Hmisc::spower… I’m surprised that there was even such a function (however obtusely named) - why on earth isn’t it in the survival library?
I was going to try to replicate that estimate, but looking at the spower documentation, it’s pretty complex and I don’t think I could do it without the original paper (which is more work than I want to do).
It is of course very difficult to extract any precise numbers from a political discussion. :) However, if you click through some of the links in the article, or have a look at the followup from today, you’ll find McArdle quoting predictions of tens of thousands of preventable deaths yearly from non-insured status. That looks to me like a pretty big hazard rate, no?
you’ll find McArdle quoting predictions of tens of thousands of preventable deaths yearly from non-insured status. That looks to me like a pretty big hazard rate, no?
No. The Oracle says there’re about 50 million Americans without health insurance. The predictions you quoted refer to 18,000 or 27,000 deaths for want of insurance per year. The higher number implies only a 0.054% death rate per year, or a 3.5% death rate over 65 years (Americans over 65 automatically get insurance). This is non-negligible but hardly huge (and potentially important for all that).
The higher number implies only a 0.054% death rate per year
Eyeballing the statistics, that looks like a hazard ratio between 1.1 and 1.5 (lots of things are good predictors for mortality that you would want to control for that I haven’t; the more you add, the closer that number should get to 1.1).
I do want to point out that, while I agree with your general points, I think that unless the proponents put numerical estimates up beforehand, it’s not quite fair to assume they meant “it will be statistically significant in a sample size of N at least 95% of the time.” Even if they said that, unless they explicitly calculated N, they probably underestimated it by at least one order of magnitude. (Professional researchers in social science make this mistake very frequently, and even when they avoid it, they can only very rarely find funding to actually collect N samples.)
I haven’t looked into this study in depth, so semi-related anecdote time: there was recently a study of calorie restriction in monkeys which had ~70 monkeys. The confidence interval for the hazard ratio included 1 (no effect), and so they concluded no statistically significant benefit to CR on mortality, though they could declare statistically significant benefit on a few varieties of mortality and several health proxies.
I ran the numbers to determine the power; turns out that they couldn’t have reliably noticed the effects of smoking (hazard ratio ~2) on longevity with a study of ~70 monkeys, and while I haven’t seen many quoted estimates of the hazard ratio of eating normally compared to CR, I don’t think there are many people that put them higher than 2.
When you don’t have the power to reliably conclude that all-cause mortality decreased, you can eke out some extra information by looking at the signs of all the proxies you measured. If insurance does nothing, we should expect to see the effect estimates scattered around 0. If insurance has a positive effect, we should expect to see more effect estimates above 0 than below 0, even though most will include 0 in their CI. (Suppose they measure 30 mortality proxies, and all of them show a positive effect, though the univariate CI includes 0 for all of them. If the ground truth was no effect on mortality proxies, that’s a very unlikely result to see; if the ground truth was a positive effect on mortality proxies, that’s a likely result to see.)
Incidentally, how did you do that?
If I remember correctly, I noticed an effect that did give a p of slightly less than .05 was a hazard ratio of 3, which made me think of running that test, and then I think spower was the r function that I used to figure out what p they could get for a hazard ratio of 2 and 35 experimentals and 35 controls (or whatever the actual split was- I think it was slightly different?).
So you were using
Hmisc::spower
… I’m surprised that there was even such a function (however obtusely named) - why on earth isn’t it in thesurvival
library?I was going to try to replicate that estimate, but looking at the spower documentation, it’s pretty complex and I don’t think I could do it without the original paper (which is more work than I want to do).
It is of course very difficult to extract any precise numbers from a political discussion. :) However, if you click through some of the links in the article, or have a look at the followup from today, you’ll find McArdle quoting predictions of tens of thousands of preventable deaths yearly from non-insured status. That looks to me like a pretty big hazard rate, no?
No. The Oracle says there’re about 50 million Americans without health insurance. The predictions you quoted refer to 18,000 or 27,000 deaths for want of insurance per year. The higher number implies only a 0.054% death rate per year, or a 3.5% death rate over 65 years (Americans over 65 automatically get insurance). This is non-negligible but hardly huge (and potentially important for all that).
Edit: and I see gwern has whupped me here.
Eyeballing the statistics, that looks like a hazard ratio between 1.1 and 1.5 (lots of things are good predictors for mortality that you would want to control for that I haven’t; the more you add, the closer that number should get to 1.1).
It looks like you’re referring to a hazard ratio or maybe a relative risk, neither of which are the same as a “hazard rate” AFAIK.
You’re right; I’m thinking of hazard ratios. Editing.
Over a population of something like 50 million people? Dunno.