This may be true, but McArdle’s point is precisely that this was not said before the study came out. At that time, people confidently expected that health insurance would, in fact, improve health outcomes. Your argument is one that was only made after the result was known; this is a classic failure mode.
(nods) Yup. Of course, McArdle’s claims about what people would have said before the study, if asked, are also only being made after the results are known, which as you say is a classic failure mode.
Of course, McArdle is neither passing laws nor doing research, just writing articles, so the cost of failure is low. And it’s kind of nice to see someone in the mainstream (sorta) press making the point that surprising observations should change our confidence in our beliefs, which people surprisingly often overlook.
Anyway, the quality of McArdle’s analysis notwithstanding, one place this sort of reasoning seems to lead us is to the idea that when passing a law, we ought to say something about what we anticipate the results of passing that law to be, and have a convention of repealing laws that don’t actually accomplish the thing that we said we were passing the law in order to accomplish.
Which in principle I would be all in favor of, except for the obvious failure mode that if I personally don’t want us to accomplish that, I am now given an incentive to manipulate the system in other ways to lower whatever metrics we said we were going to measure. (Note: I am not claiming here that any such thing happened in the Oregon study.)
That said, even taking that failure mode into account, it might still be preferable to passing laws with unarticulated expected benefits and keeping them on the books despite those benefits never materializing.
Of course, McArdle’s claims about what people would have said before the study, if asked, are also only being made after the results are known, which as you say is a classic failure mode.
I don’t think that’s true; if you read her original article on the subject, linked in the one I link, she quotes statistics like this:
Most of you probably have probably heard the statistic that being uninsured kills 18,000 people a year. Or maybe it’s 27,000. Those figures come from an Institute of Medicine report (later updated by the Urban Institute) that was drawn from [nonrandom observational] studies.
I took a keen interest when, at the fervid climax of the health-care debate in mid-December, a Washington Post blogger, Ezra Klein, declared that Senator Joseph Lieberman, by refusing to vote for a bill with a public option, was apparently “willing to cause the deaths of hundreds of thousands” of uninsured people in order to punish the progressives who had opposed his reelection in 2006. In the ensuing blogstorm, conservatives condemned Klein’s “venomous smear,” while liberals solemnly debated the circumstances under which one may properly accuse one’s opponents of mass murder.
Fair enough. I only read the article you linked, not the additional source material; I’m prepared to believe given additional evidence like what you cite here that her analysis is… er… can one say “pre-hoc”?
[W]hen passing a law, we ought to say something about what we anticipate the results of passing that law to be, and have a convention of repealing laws that don’t actually accomplish the thing that we said we were passing the law in order to accomplish.
There would have to be a two sided test. A tort of ineffectiveness by which the plaintiff seeks relief from a law that fails to achieve the goals laid out for it. A tort of under-ambition by which the plaintiff seeks relief from a law that is immune from the tort of ineffectiveness because the formally specified goals are feeble.
Think about the American experience with courts voiding laws that are unconstitutional. This often ends up with the courts applying balancing tests. It can end up with the court ruling that yes, the law infringes your rights, but only a little. And the law serves a valid purpose, which is very important. So the law is allowed to stand.
These kinds of cases are decided in prospect. The decision is reached on the speculation about the actual effects of the law. It might help if constitutional challenges to legislation could be re-litigated, perhaps after the first ten years. The second hearing could then be decided retrospectively, looking back at ten years experience, and balancing the actual burden on the plaintiffs rights against the actual public benefit of the law.
Where though is the goal post? In practice it moves. In the prospective hearing the government will make grand promises about the huge benefits the law will bring. In the retrospective hearing the government will sail on the opposite tack, arguing that only very modest benefits suffice to justify the law.
It would be good it the goal posts are fixed. Right from the start the law states the goals against which it will be assessed in ten years time. Certainly there needs to be a tort of ineffectiveness, active against laws that do not meet their goals. But politicians would soon learn to game the system by writing very modest goals into law. That needs to be blocked with a tort of under-ambition which ensures that the initial constitutionality of the law is judged only admitting in prospect those benefits that can be litigated in retrospect.
The goal posts should definitely be fixed! And maybe some politicians would want to pass a law that benefits him and his friends in some way, even though it only has a small effect, so there ought to be some kind of safeguard against that, too. But the main problem I can see is anti-synergy. Suppose a law is adopted that totally would have worked, were it not for some other law that was introduced a little later? Should the first one be repealed, or the second one? But maybe the second one does accomplish its goal, and repealing the first one would have negative effects, now that the second one is in place… And with so many laws interacting, how can you even tell which ones have which effects, unless the effects are very large indeed? (Of course, this is a problem in the current system too. I’m glad I’m not a politician; I’d be paralyzed with fear of unintended consequences.)
This is a perspective similar to DanielLC’s point. Additionally, a commenter there makes the parallel point that we don’t really know whether private insurance improves the outcome measures.
Your argument is one that was only made after the result was known; this is a classic failure mode.
True, but we shouldn’t overstate the argument. The p-values were not low enough to count as “statistically significant,” but the direction of change was towards improved health outcomes. One is doing something wrong with this evidence if one updates against improved health outcomes for public health insurance for the poor (i.e. Medicaid).
One is doing something wrong with this evidence if one updates against improved health outcomes for public health insurance for the poor (i.e. Medicaid).
Updates always move you towards what you just saw, and so if your estimate was above what you just saw, you update down. If you only consider the hypotheses that Medicaid “improves,” “has no effect,” or “harms,” then this is weak evidence for “improves” (and “has no effect”). But a more sophisticated set of hypotheses is the quantitative effect of Medicaid; if one estimated beforehand that Medicaid doubled lifespans (to use an exaggerated example), they should revise their estimate downward after seeing this study.
Fair enough. I should have said “McArdle and her political allies are making a mistake by not updating towards ‘Medicaid improves health outcomes,’” given my perception of their priors.
This may be true, but McArdle’s point is precisely that this was not said before the study came out. At that time, people confidently expected that health insurance would, in fact, improve health outcomes. Your argument is one that was only made after the result was known; this is a classic failure mode.
(nods) Yup. Of course, McArdle’s claims about what people would have said before the study, if asked, are also only being made after the results are known, which as you say is a classic failure mode.
Of course, McArdle is neither passing laws nor doing research, just writing articles, so the cost of failure is low. And it’s kind of nice to see someone in the mainstream (sorta) press making the point that surprising observations should change our confidence in our beliefs, which people surprisingly often overlook.
Anyway, the quality of McArdle’s analysis notwithstanding, one place this sort of reasoning seems to lead us is to the idea that when passing a law, we ought to say something about what we anticipate the results of passing that law to be, and have a convention of repealing laws that don’t actually accomplish the thing that we said we were passing the law in order to accomplish.
Which in principle I would be all in favor of, except for the obvious failure mode that if I personally don’t want us to accomplish that, I am now given an incentive to manipulate the system in other ways to lower whatever metrics we said we were going to measure. (Note: I am not claiming here that any such thing happened in the Oregon study.)
That said, even taking that failure mode into account, it might still be preferable to passing laws with unarticulated expected benefits and keeping them on the books despite those benefits never materializing.
I don’t think that’s true; if you read her original article on the subject, linked in the one I link, she quotes statistics like this:
And back in 2010, she said
I don’t think her statement is entirely post-hoc.
Fair enough. I only read the article you linked, not the additional source material; I’m prepared to believe given additional evidence like what you cite here that her analysis is… er… can one say “pre-hoc”?
Ante hoc.
Well, if not, one ought to be able to. I hereby grant you permission! :)
I love this idea!
There would have to be a two sided test. A tort of ineffectiveness by which the plaintiff seeks relief from a law that fails to achieve the goals laid out for it. A tort of under-ambition by which the plaintiff seeks relief from a law that is immune from the tort of ineffectiveness because the formally specified goals are feeble.
Think about the American experience with courts voiding laws that are unconstitutional. This often ends up with the courts applying balancing tests. It can end up with the court ruling that yes, the law infringes your rights, but only a little. And the law serves a valid purpose, which is very important. So the law is allowed to stand.
These kinds of cases are decided in prospect. The decision is reached on the speculation about the actual effects of the law. It might help if constitutional challenges to legislation could be re-litigated, perhaps after the first ten years. The second hearing could then be decided retrospectively, looking back at ten years experience, and balancing the actual burden on the plaintiffs rights against the actual public benefit of the law.
Where though is the goal post? In practice it moves. In the prospective hearing the government will make grand promises about the huge benefits the law will bring. In the retrospective hearing the government will sail on the opposite tack, arguing that only very modest benefits suffice to justify the law.
It would be good it the goal posts are fixed. Right from the start the law states the goals against which it will be assessed in ten years time. Certainly there needs to be a tort of ineffectiveness, active against laws that do not meet their goals. But politicians would soon learn to game the system by writing very modest goals into law. That needs to be blocked with a tort of under-ambition which ensures that the initial constitutionality of the law is judged only admitting in prospect those benefits that can be litigated in retrospect.
The goal posts should definitely be fixed! And maybe some politicians would want to pass a law that benefits him and his friends in some way, even though it only has a small effect, so there ought to be some kind of safeguard against that, too. But the main problem I can see is anti-synergy. Suppose a law is adopted that totally would have worked, were it not for some other law that was introduced a little later? Should the first one be repealed, or the second one? But maybe the second one does accomplish its goal, and repealing the first one would have negative effects, now that the second one is in place… And with so many laws interacting, how can you even tell which ones have which effects, unless the effects are very large indeed? (Of course, this is a problem in the current system too. I’m glad I’m not a politician; I’d be paralyzed with fear of unintended consequences.)
Good point! I’ve totally failed to think about multiple laws interacting.
This is a perspective similar to DanielLC’s point. Additionally, a commenter there makes the parallel point that we don’t really know whether private insurance improves the outcome measures.
True, but we shouldn’t overstate the argument. The p-values were not low enough to count as “statistically significant,” but the direction of change was towards improved health outcomes. One is doing something wrong with this evidence if one updates against improved health outcomes for public health insurance for the poor (i.e. Medicaid).
Updates always move you towards what you just saw, and so if your estimate was above what you just saw, you update down. If you only consider the hypotheses that Medicaid “improves,” “has no effect,” or “harms,” then this is weak evidence for “improves” (and “has no effect”). But a more sophisticated set of hypotheses is the quantitative effect of Medicaid; if one estimated beforehand that Medicaid doubled lifespans (to use an exaggerated example), they should revise their estimate downward after seeing this study.
Fair enough. I should have said “McArdle and her political allies are making a mistake by not updating towards ‘Medicaid improves health outcomes,’” given my perception of their priors.