Beware The Man Of One Study

Aquinas fa­mously said: be­ware the man of one book. I would add: be­ware the man of one study.

For ex­am­ple, take med­i­cal re­search. Sup­pose a cer­tain drug is weakly effec­tive against a cer­tain dis­ease. After a few years, a bunch of differ­ent re­search groups have got­ten their hands on it and done all sorts of differ­ent stud­ies. In the best case sce­nario the av­er­age study will find the true re­sult – that it’s weakly effec­tive.

But there will also be ran­dom noise caused by in­evitable vari­a­tion and by some of the ex­per­i­ments be­ing bet­ter qual­ity than oth­ers. In the end, we might ex­pect some­thing look­ing kind of like a bell curve. The peak will be at “weakly effec­tive”, but there will be a few stud­ies to ei­ther side. Some­thing like this:

We see that the peak of the curve is some­where to the right of neu­tral – ie weakly effec­tive – and that there are about 15 stud­ies that find this cor­rect re­sult.

But there are also about 5 stud­ies that find that the drug is very good, and 5 stud­ies miss­ing the sign en­tirely and find­ing that the drug is ac­tively bad. There’s even 1 study find­ing that the drug is very bad, maybe se­ri­ously dan­ger­ous.

This is be­fore we get into fraud or statis­ti­cal malprac­tice. I’m say­ing this is what’s go­ing to hap­pen just by nor­mal vari­a­tion in ex­per­i­men­tal de­sign. As we in­crease ex­per­i­men­tal rigor, the bell curve might get squashed hori­zon­tally, but there will still be a bell curve.

In prac­tice it’s worse than this, be­cause this is as­sum­ing ev­ery­one is in­ves­ti­gat­ing ex­actly the same ques­tion.

Sup­pose that the graph is ti­tled “Effec­tive­ness Of This Drug In Treat­ing Bipo­lar Di­sor­der”.

But maybe the drug is more effec­tive in bipo­lar i than in bipo­lar ii (Depakote, for ex­am­ple)

Or maybe the drug is very effec­tive against bipo­lar ma­nia, but much less effec­tive against bipo­lar de­pres­sion (Depakote again).

Or maybe the drug is a good acute an­ti­manic agent, but very poor at main­te­nance treat­ment (let’s stick with Depakote).

If you have a graph ti­tled “Effec­tive­ness Of Depakote In Treat­ing Bipo­lar Di­sor­der” plot­ting stud­ies from “Very Bad” to “Very Good” – and you stick all the stud­ies – main­te­nence, manic, de­pres­sive, bipo­lar i, bipo­lar ii – on the graph, then you’re go­ing to end run­ning the gamut from “very bad” to “very good” even be­fore you fac­tor in noise and even be­fore even be­fore you fac­tor in bias and poor ex­per­i­men­tal de­sign.

So here’s why you should be­ware the man of one study.

If you go to your bet­ter class of al­ter­na­tive medicine web­sites, they don’t tell you “Stud­ies are a lo­go­cen­tric phal­lo­cen­tric tool of Western medicine and the Big Pharma con­spir­acy.”

They tell you “med­i­cal sci­ence has proved that this drug is ter­rible, but ig­no­rant doc­tors are push­ing it on you any­way. Look, here’s a study by a rep­utable in­sti­tu­tion prov­ing that the drug is not only in­effec­tive, but harm­ful.”

And the study will ex­ist, and the au­thors will be pres­ti­gious sci­en­tists, and it will prob­a­bly be about as rigor­ous and well-done as any other study.

And then a lot of peo­ple raised on the idea that some things have Ev­i­dence and other things have No Ev­i­dence think holy s**t, they’re right!

On the other hand, your doc­tor isn’t go­ing to a sketchy al­ter­na­tive medicine web­site. She’s ex­am­in­ing the en­tire liter­a­ture and ex­tract­ing care­ful and well-in­formed con­clu­sions from…

Haha, just kid­ding. She’s go­ing to a lun­cheon at a re­ally nice restau­rant spon­sored by a phar­ma­ceu­ti­cal com­pany, which as­sures her that they would never take ad­van­tage of such an op­por­tu­nity to shill their drug, they just want to raise aware­ness of the lat­est study. And the lat­est study shows that their drug is great! Su­per great! And your doc­tor nods along, be­cause the au­thors of the study are pres­ti­gious sci­en­tists, and it’s about as rigor­ous and well-done as any other study.

But ob­vi­ously the phar­ma­ceu­ti­cal com­pany has se­lected one of the stud­ies from the “very good” end of the bell curve.

And I called this “Be­ware The Man of One Study”, but it’s easy to see that in the lit­tle di­a­gram there are like three or four stud­ies show­ing that the drug is “very good”, so if your doc­tor is a lit­tle skep­ti­cal, the phar­ma­ceu­ti­cal com­pany can say “You are right to be skep­ti­cal, one study doesn’t prove any­thing, but look – here’s an­other group that finds the same thing, here’s yet an­other group that finds the same thing, and here’s a repli­ca­tion that con­firms both of them.”

And even though it looks like in our ex­am­ple the sketchy al­ter­na­tive medicine web­site only has one “very bad” study to go off of, they could eas­ily sup­ple­ment it with a bunch of merely “bad” stud­ies. Or they could add all of those stud­ies about slightly differ­ent things. Depakote is in­effec­tive at treat­ing bipo­lar de­pres­sion. Depakote is in­effec­tive at main­te­nance bipo­lar ther­apy. Depakote is in­effec­tive at bipo­lar ii.

So just sum it up as “Smith et al 1987 found the drug in­effec­tive, yet doc­tors con­tinue to pre­scribe it any­way”. Even if you hunt down the origi­nal study (which no one does), Smith et al won’t say speci­fi­cally “Do re­mem­ber that this study is only look­ing at bipo­lar main­te­nance, which is a differ­ent topic from bipo­lar acute an­ti­manic treat­ment, and we’re not say­ing any­thing about that.” It will just be ti­tled some­thing like “Depakote fails to sep­a­rate from placebo in six month trial of 91 pa­tients” and trust that the re­spon­si­ble pro­fes­sion­als read­ing it are well aware of the differ­ence be­tween acute and main­te­nance treat­ments (ha­ha­ha­haha).

So it’s not so much “be­ware the man of one study” as “be­ware the man of any num­ber of stud­ies less than a rel­a­tively com­plete and not-cherry-picked sur­vey of the re­search”.

II.

I think med­i­cal sci­ence is still pretty healthy, and that the con­sen­sus of doc­tors and re­searchers is more-or-less right on most con­tro­ver­sial med­i­cal is­sues.

(it’s the un­con­tro­ver­sial ones you have to worry about)

Poli­tics doesn’t have this pro­tec­tion.

Like, take the min­i­mum wage ques­tion (please). We all know about the Krueger and Card study in New Jersey that found no ev­i­dence that high min­i­mum wages hurt the econ­omy. We prob­a­bly also know the coun­ter­claims that it was com­pletely de­bunked as de­spi­ca­ble dishon­est statis­ti­cal malprac­tice. Maybe some of us know Card and Krueger wrote a pretty con­vinc­ing re­but­tal of those claims. Or that a bunch of large and method­olog­i­cally ad­vanced stud­ies have come out since then, some find­ing no effect like Dube, oth­ers find­ing strong effects like Ru­bin­stein and Wither. Th­ese are just ex­am­ples; there are at least dozens and prob­a­bly hun­dreds of stud­ies on both sides.

But we can solve this with meta-analy­ses and sys­temtic re­views, right?

Depends which one you want. Do you go with this meta-anal­y­sis of four­teen stud­ies that shows that any pre­sumed nega­tive effect of high min­i­mum wages is likely pub­li­ca­tion bias? With this meta-anal­y­sis of sixty-four stud­ies that finds the same thing and dis­cov­ers no effect of min­i­mum wage af­ter cor­rect­ing for the prob­lem? Or how about this meta-anal­y­sis of fifty-five coun­tries that does find effects in most of them? Maybe you pre­fer this sys­tem­atic re­view of a hun­dred or so stud­ies that finds strong and con­sis­tent effects?

Can we trust news sources, think tanks, econ­blogs, and other in­sti­tu­tions to sum up the state of the ev­i­dence?

CNN claims that 85% of cred­ible stud­ies have shown the min­i­mum wage causes job loss. But raisethem­i­ni­mumwage.com de­clares that “two decades of rigor­ous eco­nomic re­search have found that rais­ing the min­i­mum wage does not re­sult in job loss…re­searchers and busi­nesses al­ike agree to­day that the weight of the ev­i­dence shows no re­duc­tion in em­ploy­ment re­sult­ing from min­i­mum wage in­creases.” Modeled Be­hav­ior says “the ma­jor­ity of the new min­i­mum wage re­search sup­ports the hy­poth­e­sis that the min­i­mum wage in­creases un­em­ploy­ment.” The Cen­ter for Bud­get and Policy Pri­ori­ties says “The com­mon claim that rais­ing the min­i­mum wage re­duces em­ploy­ment for low-wage work­ers is one of the most ex­ten­sively stud­ied is­sues in em­piri­cal eco­nomics. The weight of the ev­i­dence is that such im­pacts are small to none.”

Okay, fine. What about economists? They seem like ex­perts. What do they think?

Well, five hun­dred economists signed a let­ter to policy mak­ers say­ing that the sci­ence of eco­nomics shows in­creas­ing the min­i­mum wage would be a bad idea. That sounds like a promis­ing con­sen­sus…

..ex­cept that six hun­dred economists signed a let­ter to policy mak­ers say­ing that the sci­ence of eco­nomics shows in­creas­ing the min­i­mum wage would be a good idea. (h/​t Greg Mankiw)

Fine then. Let’s do a for­mal sur­vey of economists. Now what?

raisethem­i­ni­mumwage.com, an un­bi­ased source if ever there was one, con­fi­dently tells us that<A> “in­dica­tive is a 2013 sur­vey by the Univer­sity of Chicago’s Booth School of Busi­ness in which lead­ing economists agreed by a nearly 4 to 1 mar­gin that the benefits of rais­ing and in­dex­ing the min­i­mum wage out­weigh the costs.”

But the Em­ploy­ment Poli­cies In­sti­tute, which sounds like it’s try­ing way too hard to sound like an un­bi­ased source, tells us that “Over 73 per­cent of AEA la­bor economists be­lieve that a sig­nifi­cant in­crease will lead to em­ploy­ment losses and 68 per­cent think these em­ploy­ment losses fall dis­pro­por­tionately on the least skil­led. Only 6 per­cent feel that min­i­mum wage hikes are an effi­cient way to alle­vi­ate poverty.”

So the whole thing is fiendishly com­pli­cated. But un­less you look very very hard, you will never know that.

If you are a con­ser­va­tive, what you will find on the sites you trust will be some­thing like this:

Eco­nomic the­ory has always shown that min­i­mum wage in­creases de­crease em­ploy­ment, but the Left has never been will­ing to ac­cept this ba­sic fact. In 1992, they trum­peted a sin­gle study by Card and Krueger that pur­ported to show no nega­tive effects from a min­i­mum wage in­crease. This study was im­me­di­ately de­bunked and found to be based on statis­ti­cal malprac­tice and “mas­sag­ing the num­bers”. Since then, dozens of stud­ies have come out con­firm­ing what we knew all along – that a high min­i­mum wage is eco­nomic suicide. Sys­tem­atic re­views and meta-analy­ses (Neu­mark 2006, Boock­man 2010) con­sis­tently show that an over­whelming ma­jor­ity of the re­search agrees on this fact – as do 73% of economists. That’s why five hun­dred top economists re­cently signed a let­ter urg­ing policy mak­ers not to buy into dis­cred­ited liberal min­i­mum wage the­o­ries. In­stead of listen­ing to starry-eyed liberal woo, listen to the em­piri­cal ev­i­dence and an over­whelming ma­jor­ity of economists and op­pose a raise in the min­i­mum wage.

And if you are a leftist, what you will find on the sites you trust will be some­thing like this:

Peo­ple used to be­lieve that the min­i­mum wage de­creased un­em­ploy­ment. But Card and Krueger’s fa­mous 1992 study ex­ploded that con­ven­tional wis­dom. Since then, the re­sults have been repli­cated over fifty times, and fur­ther meta-analy­ses (Card and Krueger 1995, Dube 2010) have found no ev­i­dence of any effect. Lead­ing economists agree by a 4 to 1 mar­gin that the benefits of rais­ing the min­i­mum wage out­weigh the costs, and that’s why more than 600 of them have signed a pe­ti­tion tel­ling the gov­ern­ment to do ex­actly that. In­stead of listen­ing to con­ser­va­tive scare tac­tics based on long-de­bunked the­o­ries, listen to the em­piri­cal ev­i­dence and the over­whelming ma­jor­ity of economists and sup­port a raise in the min­i­mum wage.

Go ahead. Google the is­sue and see what stuff comes up. If it doesn’t quite match what I said above, it’s usu­ally be­cause they can’t even muster that level of schol­ar­ship. Half the sites just cite Card and Krueger and call it a day!

Th­ese sites with their long lists of stud­ies and ex­perts are su­per con­vinc­ing. And half of them are wrong.

At some point in their ed­u­ca­tion, most smart peo­ple usu­ally learn not to credit ar­gu­ments from au­thor­ity. If some­one says “Believe me about the min­i­mum wage be­cause I seem like a trust­wor­thy guy,” most of them will have at least one neu­ron in their head that says “I should ask for some ev­i­dence”. If they’re re­ally smart, they’ll use the magic words “peer-re­viewed ex­per­i­men­tal stud­ies.”

But I worry that most smart peo­ple have not learned that a list of dozens of stud­ies, sev­eral meta-analy­ses, hun­dreds of ex­perts, and ex­pert sur­veys show­ing al­most all aca­demics sup­port your the­sis – can still be bul­lshit.

Which is too bad, be­cause that’s ex­actly what peo­ple who want to bam­boo­zle an ed­u­cated au­di­ence are go­ing to use.

III.

I do not want to preach rad­i­cal skep­ti­cism.

For ex­am­ple, on the min­i­mum wage is­sue, I no­tice only one side has pre­sented a fun­nel plot. A fun­nel plot is usu­ally used to in­ves­ti­gate pub­li­ca­tion bias, but it has an­other use as well – it’s pretty much an ex­act pre­sen­ta­tion of the “bell curve” we talked about above.

This is more of a nee­dle curve than a bell curve, but the point still stands. We see it’s cen­tered around 0, which means there’s some ev­i­dence that’s the real sig­nal among all this noise. The bell skews more to left than to the right, which means more stud­ies have found nega­tive effects of the min­i­mum wage than pos­i­tive effects of the min­i­mum wage. But since the bell curve is asym­met­ri­cal, we in­tepret that as prob­a­bly pub­li­ca­tion bias. So all in all, I think there’s at least some ev­i­dence that the liber­als are right on this one.

Un­less, of course, some­one has re­al­ized that I’ve wised up to the stud­ies and meta-analy­ses and and ex­pert sur­veys, and figured out a way to hack fun­nel plots, which I am to­tally not rul­ing out.

(okay, I kind of want to preach rad­i­cal skep­ti­cism)

Also, I should prob­a­bly men­tion that it’s much more com­pli­cated than one side be­ing right, and that the min­i­mum wage prob­a­bly works differ­ently de­pend­ing on what in­dus­try you’re talk­ing about, whether it’s state wage or fed­eral wage, whether it’s a re­ces­sion or a boom, whether we’re talk­ing about in­creas­ing from $5 to $6 or from $20 to $30, etc, etc, etc. There are eleven stud­ies on that plot show­ing an effect even worse than −5, and very pos­si­bly they are all ac­cu­rate for what­ever sub­prob­lem they have cho­sen to study – much like the ex­am­ple with Depakote where it might an effec­tive an­ti­manic but a ter­rible an­tide­pres­sant.

(rad­i­cal skep­ti­cism ac­tu­ally sounds a lot bet­ter than figur­ing this all out).

IV.

But the ques­tion re­mains: what hap­pens when (like in most cases) you don’t have a fun­nel plot?

I don’t have a good pos­i­tive an­swer. I do have sev­eral good nega­tive an­swers.

De­crease your con­fi­dence about most things if you’re not sure that you’ve in­ves­ti­gated ev­ery piece of ev­i­dence.

Do not trust web­sites which are ob­vi­ously bi­ased (eg Free Repub­lic, Daily Kos, Dr. Oz) when they tell you they’re go­ing to give you “the state of the ev­i­dence” on a cer­tain is­sue, even if the ev­i­dence seems very stately in­deed. This goes dou­ble for any site that con­tains a list of “myths and facts about X”, quadru­ple for any site that uses phrases like “in­group mem­ber uses ac­tual FACTS to DEMOLISH the out­group’s lies about Y”, and oc­tu­ple for Ra­tion­alWiki.

Most im­por­tant, even if some­one gives you what seems like over­whelming ev­i­dence in fa­vor of a cer­tain point of view, don’t trust it un­til you’ve done a sim­ple Google search to see if the op­po­site side has equally over­whelming ev­i­dence.