Thanks but doesn’t pace of adoption break the analogy? Seems safe to say that those technologies caused growth, and so will AI. But AI has been rolled out fast enough that you might have seen near-term changes in the aggregate statistics, in contrast to physical or legal infrastructure. We do have careful evidence on railroads and the printing press though—is it wild to think that an information technology spreading some large multiple faster than the printing press would have impacts in the following year? Or is it that it’s not such a technology shock at all? (Also, where do you think the education went?)
braces
Karma: 486
Great point...are there certain consistent families of GPT-4-era hallucinations that you think would be especially important in this context?
I don’t think this part is so bad. You can lack “real control” but still have an informative experiment. You just need that people in the treatment group were on average more likely to receive the treatment. The issue is how they analyzed the data.
It’s a good question. I have the intuition that just a little potential for bias can go a long way toward messing up the estimated effect, so allowing this practice is net negative despite the gains in power. The dropouts might be similar on demographics but not something unmeasured like motivation. My view comes from seeing many failed replications and playing with datasets when available, but I would love to be able to quantify this issue somehow...I would certainly predict that studies where the per protocol finding differs from the ITT will be far less likely to replicate.
I don’t have a sense of the overall prevalence, I’m curious about that too! I’ve just seen it enough in high-profile medical studies to think it’s still a big problem.
Yes this is totally related to two-stage least squares regression! The intent-to-treat estimate just gives you the effect of being assigned to treatment. The TSLS estimate scales up the intent-to-treat by the effect that the randomization had on treatment (so, e.g., if the randomization increased the share doing yoga from 10 in the control group to 50% in the treatment group, the intent-to-treat effect divided by 0.40 would give you the TSLS estimate).
FWIW, here’s a Cochrane review on RCTs of such encouragement designs. It’s basically a failed meta-analysis in that the data is too spotty to make any conclusions. But it shows at the very least that such studies have been done before. An encouragement design in a population with high alcohol use seems most promising for figuring out the causal effect on fetuses, as you say.
(Another recent meta-analysis on the same question finds a slight decrease in preterm birth in the alcohol education group, but this is based on just three studies and marginally significant so I’d say it’s still uncertain.)
Within years, the combo of ST_CASE and STATE uniquely identifies accidents, so an intermediate step is tracking the accident ID. Then you could classify accidents as involving a child as those where at least one of the decedents is under a certain age, then use that PER_TYP variable to look at the ages of the driver(s) for those accidents. Cool analysis, btw—I’ve worked a lot with the FARS and happy to help :)
You can get the age of the driver by looking at the age of the person with PER_TYP equal to one.
Super useful, I would love to read more of your IVF thoughts if they’re posted somewhere! I think your views imply that these new sperm obstacle courses, based on fears of a worse sperm-selection process in IVF, have zero effect on people’s eventual outcomes (conditional on viability)? But curious what you think. Example: “The SPARTAN system uses a series of obstacles on a microchip, requiring sperm to swim around pillar-shaped objects and through the device. As a result, this device promotes the collection of the highest quality sperm.”
Great question, I really want to know the answer! This study says “13% of non-gravid [not pregnant] women report fear of childbirth sufficient to postpone or avoid pregnancy.” Separately, roughly 15% of women never have children, so that puts a ceiling on how big the effects could be on having any children. I suspect fear of childbirth is not a top reason for the 15%...but that’s not based on much data.
Really agree with all of these, thanks. Curious, in your decision-making process, did you ever have a way to calculate “the chance of a really disabling (as bad as Down syndrome) disorder”?
Ah yes you’re right, it’s stricter for contacts. I was at least able to get to the purchase page on LensCrafters for a pair of glasses. (Just had to check a box where I promise the values are based on a recent prescription.) Thanks for pointing that out!
Thanks for the question. I like the EMH metaphor. I think that the “uncontroversially good” legislative opportunities can generally be viewed as the result of some inefficiency.
You bring up the case of diffuse harm and concentrated benefits. It seems widely acknowledged that lobbyists and interest groups have too much leverage. The inefficiency is that voters can’t keep track of all the small ways they’re being harmed and so donations do not track welfare impacts. But, as you say, these reforms would be controversial to someone, so perhaps I could improve my language. I want a pithy way to say: “Behind closed doors, most politicians would see this as utterly reasonable and good.”
While I’m on it, here are two other sources of inefficiency which I think could be relevant here:
Myopia. In the US, Social Security will be insolvent in 2033. But it seems like politicians (and voters) are not that excited about tackling this one. Pandemic preparedness is another example.
“Laboratories of democracy.” State and local governments should probably try more things than they’re selfishly incentivized to because their experience with the reforms becomes a public good that other governments can learn from and imitate.
US companies like LensCrafters, Warby Parker, and 1-800-contacts do care, which in my experience has been prohibitive in some situations. Eg see: https://www.1800contacts.com/connect/articles/buy-contact-lenses-online-without-prescription
And the discussion here: https://ask.metafilter.com/151294/Is-there-some-way-to-get-contact-lenses-when-your-prescription-is-expired
Really useful post, thanks for sharing. In case you’re interested, this paper seems to do a good job of tacking the “mortality vs years of life lost” issue you mention.
I didn’t do much research into other cross-country crime measures so I’m definitely curious to learn more about other ways of getting at this.
But to me the most obvious reason to favor homicide is reporting. Partially because it’s fresh in my mind, I’d recommend The Challenge of Crime in a Free Society (from 1967). Starting at page 25 they go through changes in reporting that caused huge blips in measured crime in the US. I think the authors may have wanted to play down the crime increase for political reasons, but even discounting them somewhat they do identify issues in measurement that could easily persist today and be way worse in other countries.
I really want to try this...From a few minutes of googling it seems pretty hard to find this info for other countries. Will update if I’m able to cobble the data together.
That makes sense. So I guess a correction where we assumed all countries had US-level healthcare would scoot the poorer countries to the left in the scatter plot...I think this could dampen trendline but would still leave the US looking pretty weird.
Good question—my reasoning was that homicide is well-measured and easy to find online, and should correlate with levels of crime more broadly. I wonder what variable we would want to address the Thailand issue...maybe the share of prisoners who are “violent” would be useful, but I’d be surprised if I could find it for many countries.
Thanks, I agree that I should better understand why so many medical researchers do this. And I can definitely imagine situations where it helps: small sample (so ITT is under powered), useful treatment (so we’re worried about false negatives), low chance of bias (compliance isn’t correlated with outcomes). My belief is that these three factors don’t align enough to justify per protocol analysis. It’s way more likely to lead you to false cures. It’s hard to test this empirically though so if you had examples of treatments that got correctly assessed and advanced with a per protocol analysis I’d be very interested!