I didn’t bother reading Drum’s article the first time I saw it circulating. I’ve known about Nevin’s papers for a couple of years, already decided they were interesting but weak evidence, and Drum didn’t seem to be bringing much new to the table. But I’ve now looked at his article, and it does reference stuff I wasn’t aware of, like the city-level correlations between vehicular lead emissions and assaults.
We now have studies based on MRI scans, ecological correlations at multiple levels of aggregation, and longitudinal studies of individuals. But they’re still all observational, and could still be affected by individual-level confounding factors. Qualitatively the causal mechanism is obviously real,* but it’s only backed up by actual experiments in vivo for blood levels of 10-50 μg/dL (as far as I know). Drum’s aiming an order of magnitude lower, where I’d still expect some effect, but I don’t trust observational studies or simple extrapolation to estimate it precisely.
The very very obvious thing to do now is run a nice, big RCT. There are RCTs for lesser interventions (e.g.) but I see none for window replacement or soil cleanup.
So: you select 100 city neighbourhoods. Send an army of researchers to each of them to bang on every door and recruit as many families with infants & toddlers as they can. Send a second wave of researchers to measure the soil in the recruited kids’ houses, take blood samples, administer the relevant IQ/development tests (test the parents too if you’re feeling hardcore), and write everyone’s demographic details (sex, age, race, etc.) down on their clipboards. Now you can choose 50 neighbourhoods at random as the treatment group; don’t forget to check it’s statistically comparable to the control group!
Having done that, get another load of people to bang on every door in the 50 neighbourhoods in the treatment arm, and offer them money to replace their windows and their soil. (I don’t know whether you can get away with not replacing windows & soil for the residents who don’t have kids in the study. If you could skip them you’d save a lot of money.) Then do the cleanup jobs.
That’s not even the hard part. You now have to keep track of all the families you’ve recruited for the next 3 decades. A lot of them will move elsewhere. (Boosting the value of their houses by eliminating the lead might even encourage them to sell up.) That’s fine — you just pay some academics to keep track of them. A year down the line, visit all of the families and run the tests again to measure the short-term changes. Some families will refuse the follow-up visit. That’s OK too — you just keep their details on file, ’cause another 5 years later, when the kids are in school, you’re gonna do a third wave of testing. Then a few more years later, when the kids are hitting puberty, you’re gonna do a fourth wave of testing, and you’re gonna get their criminal records from the cops. And again, a decade later, once they’re adults, you do the testing and criminal record check again. And you might do another one down the line just to check there aren’t any late-breaking effects to bite you. At each step, you compare the control & treatment subjects and publish the results.
How much might this all cost? Say there are 100 families’ houses to clean in each of the 50 treatment neighbourhoods. That’s 5000 houses. Drum says that Nevin says that replacing 16 million houses’ windows would cost about $200 billion in total, or $12500 per house. Soil cleanup costs about as much again, making the cleanup total $25000 per house, or $125 million for the 5000 treatment houses. Add the cost of the blood testing, paying a few academics to run the study for decades, and other odds & sods, and the total cost might be something like $150 million. Oh yeah, and your study still wouldn’t give you an unbiased estimate of the benefits of lead abatement; it’d underestimate them because of the time the children spent being exposed to lead in their houses before you showed up to run the study. But it’d give you a robust answer to a $400 billion question, and you could get part of that answer by the end of the decade.
Having done that, get another load of people to bang on every door in the 50 neighbourhoods in the treatment arm, and offer them money to replace their windows and their soil. (I don’t know whether you can get away with not replacing windows & soil for the residents who don’t have kids in the study. If you could skip them you’d save a lot of money.) Then do the cleanup jobs.
A study sort of like this was done in Rochester, and they found that nothing they did changed blood lead levels very much and so they didn’t learn anything from it. I guess they could go further with actually replacing people’s houses.
One form of more convincing evidence based on observational longitudinal data is using g-computation to adjust for the so called “time varying confounders” of lead exposure.
The paper is 120 pages, but the short version is, in graphical terms all you do is pretend that you are interested in lead exposure interventions (via do(.)) at every time slice, and simply identify this causal effect from the observational data you have. The trick is you can’t adjust for confounders as usual, because of this issue:
C → A1 → L → A2 → Y
Say A1, A2 are exposures to lead at two time slices, C is baseline confounders, L is an intermediate response, and Y is a final response. The issue is the usual adjustment here:
p[y|do[a1,a2]]=∫l∫cp[y|a1,a2,c,l]p[c,l]dcdl
is wrong. That’s because C, L and Y are all confounded by things we aren’t observing, and moreover if you condition on L, you open a path A1 → L <-> Y via these unobserved confounders which you do not want open. Here L is the “time-varying confounder”: for the purposes of A2 we want to adjust for it, but for the purposes of A1 we do not. This implies the above formula is actually wrong and will bias your estimate of the early lead exposure A1 on Y.
The issue here is you still might not have all the confounders at every time slice. But this kind of evidence is still far better than nothing at all (e.g. reporting correlations across 23 years).
Prediction: if you did this analysis, you would find no statistically significant effect on any scale.
Agreed. (And come to think of it, I’m underplaying things by calling this a “$210 billion question” — the $400 billion total cost of the interventions is as relevant as the estimated annual return.)
I didn’t bother reading Drum’s article the first time I saw it circulating. I’ve known about Nevin’s papers for a couple of years, already decided they were interesting but weak evidence, and Drum didn’t seem to be bringing much new to the table. But I’ve now looked at his article, and it does reference stuff I wasn’t aware of, like the city-level correlations between vehicular lead emissions and assaults.
We now have studies based on MRI scans, ecological correlations at multiple levels of aggregation, and longitudinal studies of individuals. But they’re still all observational, and could still be affected by individual-level confounding factors. Qualitatively the causal mechanism is obviously real,* but it’s only backed up by actual experiments in vivo for blood levels of 10-50 μg/dL (as far as I know). Drum’s aiming an order of magnitude lower, where I’d still expect some effect, but I don’t trust observational studies or simple extrapolation to estimate it precisely.
The very very obvious thing to do now is run a nice, big RCT. There are RCTs for lesser interventions (e.g.) but I see none for window replacement or soil cleanup.
So: you select 100 city neighbourhoods. Send an army of researchers to each of them to bang on every door and recruit as many families with infants & toddlers as they can. Send a second wave of researchers to measure the soil in the recruited kids’ houses, take blood samples, administer the relevant IQ/development tests (test the parents too if you’re feeling hardcore), and write everyone’s demographic details (sex, age, race, etc.) down on their clipboards. Now you can choose 50 neighbourhoods at random as the treatment group; don’t forget to check it’s statistically comparable to the control group!
Having done that, get another load of people to bang on every door in the 50 neighbourhoods in the treatment arm, and offer them money to replace their windows and their soil. (I don’t know whether you can get away with not replacing windows & soil for the residents who don’t have kids in the study. If you could skip them you’d save a lot of money.) Then do the cleanup jobs.
That’s not even the hard part. You now have to keep track of all the families you’ve recruited for the next 3 decades. A lot of them will move elsewhere. (Boosting the value of their houses by eliminating the lead might even encourage them to sell up.) That’s fine — you just pay some academics to keep track of them. A year down the line, visit all of the families and run the tests again to measure the short-term changes. Some families will refuse the follow-up visit. That’s OK too — you just keep their details on file, ’cause another 5 years later, when the kids are in school, you’re gonna do a third wave of testing. Then a few more years later, when the kids are hitting puberty, you’re gonna do a fourth wave of testing, and you’re gonna get their criminal records from the cops. And again, a decade later, once they’re adults, you do the testing and criminal record check again. And you might do another one down the line just to check there aren’t any late-breaking effects to bite you. At each step, you compare the control & treatment subjects and publish the results.
How much might this all cost? Say there are 100 families’ houses to clean in each of the 50 treatment neighbourhoods. That’s 5000 houses. Drum says that Nevin says that replacing 16 million houses’ windows would cost about $200 billion in total, or $12500 per house. Soil cleanup costs about as much again, making the cleanup total $25000 per house, or $125 million for the 5000 treatment houses. Add the cost of the blood testing, paying a few academics to run the study for decades, and other odds & sods, and the total cost might be something like $150 million. Oh yeah, and your study still wouldn’t give you an unbiased estimate of the benefits of lead abatement; it’d underestimate them because of the time the children spent being exposed to lead in their houses before you showed up to run the study. But it’d give you a robust answer to a $400 billion question, and you could get part of that answer by the end of the decade.
* Blood lead poisoning has been observed since antiquity; sufficiently high exposure causes death within weeks or months; lead-treated lab animals show behavioural deficits; non-fatal but high blood lead levels cause obvious symptoms in children that can be partially reversed with chelation.
A study sort of like this was done in Rochester, and they found that nothing they did changed blood lead levels very much and so they didn’t learn anything from it. I guess they could go further with actually replacing people’s houses.
The hidden agenda of Extreme House Makeover.
One form of more convincing evidence based on observational longitudinal data is using g-computation to adjust for the so called “time varying confounders” of lead exposure.
A classic paper on this from way back in 1986 is this: http://www.biostat.harvard.edu/robins/new-approach.pdf
The paper is 120 pages, but the short version is, in graphical terms all you do is pretend that you are interested in lead exposure interventions (via do(.)) at every time slice, and simply identify this causal effect from the observational data you have. The trick is you can’t adjust for confounders as usual, because of this issue:
C → A1 → L → A2 → Y
Say A1, A2 are exposures to lead at two time slices, C is baseline confounders, L is an intermediate response, and Y is a final response. The issue is the usual adjustment here:
p[y|do[a1,a2]]=∫l∫cp[y|a1,a2,c,l]p[c,l]dcdl
is wrong. That’s because C, L and Y are all confounded by things we aren’t observing, and moreover if you condition on L, you open a path A1 → L <-> Y via these unobserved confounders which you do not want open. Here L is the “time-varying confounder”: for the purposes of A2 we want to adjust for it, but for the purposes of A1 we do not. This implies the above formula is actually wrong and will bias your estimate of the early lead exposure A1 on Y.
What we want to do instead is this:
p[y|do[a1,a2]]=∫l∫cp[y|a1,a2,c,l]p[l|a1,c]p[c]dcdl
The issue here is you still might not have all the confounders at every time slice. But this kind of evidence is still far better than nothing at all (e.g. reporting correlations across 23 years).
Prediction: if you did this analysis, you would find no statistically significant effect on any scale.
For a proposed $400b intervention, a few hundred millions seem like pretty reasonable expenditures.
Agreed. (And come to think of it, I’m underplaying things by calling this a “$210 billion question” — the $400 billion total cost of the interventions is as relevant as the estimated annual return.)