Debunked And Well-Refuted

I.

As usual, I was in­suffi­ciently pes­simistic.

I in­fer this from The Fed­er­al­ist‘s ar­ti­cle on cam­pus rape:

A new re­port on sex­ual as­sault re­leased to­day by the U.S. Depart­ment of Jus­tice (DOJ) offi­cially puts to bed the bo­gus statis­tic that one in five women on col­lege cam­puses are vic­tims of sex­ual as­sault. In fact, non-stu­dents are 25 per­cent more likely to be vic­tims of sex­ual as­sault than stu­dents, ac­cord­ing to the data. And the real num­ber of as­sault vic­tims is sev­eral or­ders of mag­ni­tude lower than one-in-five.

The ar­ti­cle com­pares the older Cam­pus Sex­ual As­sault Sur­vey (which found 14-20% of women were raped since en­ter­ing col­lege) to the just-re­leased Na­tional Crime Vict­miza­tion Sur­vey (which found that 0.6% of fe­male col­lege stu­dents are raped per year). They write “In­stead of 1 in 5, the real num­ber is 0.03 in 5.”

So the first thing I will mock The Fed­er­al­ist for do­ing is di­rectly com­par­ing per year sex­ual as­sault rates to per col­lege ca­reer sex­ual as­sault rates, whereas ob­vi­ously these are very differ­ent things. You can’t quite just di­vide the lat­ter by four to get the former, but that’s go­ing to work a heck of a lot bet­ter than not do­ing it, so let’s es­ti­mate the real dis­crep­ancy as more like 0.5% per year ver­sus 5% per year.

But I can’t get too mad at them yet, be­cause that’s still a pretty big dis­crep­ancy.

How­ever, faced with this dis­crep­ancy a rea­son­able per­son might say “Hmm, we have two differ­ent stud­ies that say two differ­ent things. I won­der what’s go­ing on here and which study we should be­lieve?”

The Fed­er­al­ist staff said “Ha! There’s an old study with find­ings we didn’t like, but now there’s a new study with differ­ent find­ings we do like. So the old study is de­bunked!”

II.

My last es­say, Be­ware The Man Of One Study, noted that one thing par­ti­sans do to jus­tify their bias is se­lec­tively ac­knowl­edge stud­ies from only one side of a com­pli­cated liter­a­ture.

The rea­son it was in­suffi­ciently pes­simistic is that there are also peo­ple like the Fed­er­al­ist staff, who ac­knowl­edge the ex­is­tence of op­pos­ing stud­ies, but only with the ad­jec­tive “de­bunked” in front of them. By “de­bunked” they usu­ally mean one of two things:

1. Some­one on my side pub­lished a study later that found some­thing else
2. Some­one on my side ac­cused it of hav­ing method­olog­i­cal flaws

Since the Fed­er­al­ist has so am­ply demon­strated the first failure mode, let me say a lit­tle more about the sec­ond. Did you know that any­one with a key­board can just type up any of the fol­low­ing things?

– “That study is a piece of garbage that’s not worth the pa­per it’s writ­ten on.”
– “Peo­ple in the know dis­missed that study years ago.”
– “No­body in the field takes that study se­ri­ously.”
– “That study uses meth­ods that are laugh­able to any­body who knows statis­tics.”
– “All the other re­search that has come out since dis­cred­its that study.”

They can say these things whether they are true or not. I’m kind of harp­ing on this point, but it’s be­cause it’s some­thing I didn’t re­al­ize un­til much later than I should have.

There are many “ques­tions” that are pretty much set­tled – evolu­tion, global warm­ing, home­opa­thy. But tak­ing these as rep­re­sen­ta­tive closes your mind and gives you a skewed pic­ture of academia. On many is­sues, aca­demics are just as di­vided as any­one else, and their ar­gu­ments can be just as ac­rimo­nious as any­one else’s. The ar­gu­ments usu­ally take the form of one side pub­lish­ing a study, the other side rip­ping the study apart and pub­lish­ing their own study which they say is bet­ter, and the first side rip­ping the sec­ond study apart and ar­gu­ing that their study was bet­ter all along.

Every study has flaws. No study has perfect method­ol­ogy. If you like a study, you can say that it did the best it could on a difficult re­search area and has im­proved upon even-worse pre­de­ces­sor stud­ies. If you don’t like a study, you can say “LOOK AT THESE FLAWS THESE PEOPLE ARE IDIOTS THE CONCLUSION IS COMPLETELY INVALID”. All you need to do is make enough iso­lated de­mands for rigor against any­thing you dis­agree with.

And so if the first level of con­fir­ma­tion bias is be­liev­ing ev­ery study that sup­ports your views, the sec­ond layer of con­fir­ma­tion bias is be­liev­ing ev­ery sup­posed re­fu­ta­tion that sup­ports your views.

There are cer­tainly things that have been “well-re­futed” and “de­bunked”. An­drew Wakefield’s study pur­port­ing to prove that vac­cines cause autism is a pretty good ex­am­ple. But you will no­tice that it had mul­ti­ple failed repli­ca­tions, jour­nals pub­lished re­ports show­ing he falsified data, the study’s co-au­thors re­tracted their sup­port, the jour­nal it was pub­lished in re­tracted it and is­sued an apol­ogy, the Gen­eral Med­i­cal Coun­cil con­victed Wakefield of six­teen counts of mis­con­duct, and Wakefield was stripped of his med­i­cal li­cense and barred from prac­tic­ing medicine ever again in the UK. The Bri­tish Med­i­cal Jour­nal, one of the best-re­spected med­i­cal jour­nals in the world, pub­lished an ed­i­to­rial con­clud­ing:

Clear ev­i­dence of falsifi­ca­tion of data should now close the door on this dam­ag­ing vac­cine scare … Who per­pe­trated this fraud? There is no doubt that it was Wakefield. Is it pos­si­ble that he was wrong, but not dishon­est: that he was so in­com­pe­tent that he was un­able to fairly de­scribe the pro­ject, or to re­port even one of the 12 chil­dren’s cases ac­cu­rately? No.

Wakefield’s study has been “re­futed”. The rape study has been “ar­gued against”.

III.

I saw this same dy­namic at work the other day, look­ing through the min­i­mum wage liter­a­ture.

The pri­mor­dial titanomachy of the min­i­mum wage liter­a­ture goes like this. In 1994, two guys named Card and Krueger pub­lished a study show­ing the min­i­mum wage had if any­thing pos­i­tive effects on New Jersey restau­rants, con­vinc­ing many peo­ple that min­i­mum wages were good. In 1996, two guys named Neu­mark and Wascher re­an­a­lyzed the New Jersey data us­ing a differ­ent source and found that it showed the min­i­mum wage had very bad effects on New Jersey restau­rants. In 2000, Card and Krueger re­sponded, say­ing that their anal­y­sis was bet­ter than Neu­mark and Wascher’s re-anal­y­sis, and also they had done a re-anal­y­sis of their own which con­firmed their origi­nal po­si­tion.

Let’s see how con­ser­va­tive sites pre­sent this pic­ture:

“The sup­port for this as­ser­tion is the oft-cited 1994 study by Card and Krueger show­ing a pos­i­tive cor­re­la­tion be­tween an in­creased min­i­mum wage and em­ploy­ment in New Jersey. Many oth­ers have thor­oughly de­bunked this study.” (source)

“I was un­der the im­pres­sion that the origi­nal study done by Card and Krueger had been thor­oughly de­bunked by Michi­gan State Univer­sity economist David Neu­mark and William Wascher” (source)

“The study … by Card and Krueger has been de­bunked by sev­eral differ­ent peo­ple sev­eral differ­ent times. When other re­searchers re-eval­u­ated the study, they found that data col­lected us­ing those records ‘lead to the op­po­site con­clu­sion from that reached by’ Card and Krueger.” (source)

“It was only a short time be­fore the fan­tas­tic Card-Krueger find­ings were challenged and de­bunked by sev­eral sub­se­quent stud­ies…in 1995, economists David Neu­mark and David Wascher used ac­tual pay­roll records (in­stead of sur­vey data used by Card and Krueger) and pub­lished their re­sults in an NBER pa­per with an amaz­ing find­ing: De­mand curves for un­skil­led la­bor re­ally do slope down­ward, con­firm­ing 200 years of eco­nomic the­ory and moun­tains of em­piri­cal ev­i­dence (source)

And now let’s look at how lefty sites pre­sent this pic­ture:

“…a long-de­bunked pa­per [by Neu­mark and Wascher]” (source)

“Note that your Mises heroes, Neu­mark and Wascher are roundly de­bunked.” (source)

“Neu­mark’s liv­ing wage and min­i­mum wage re­search have been found to be se­ri­ously flawed…based on faulty meth­ods which when cor­rected re­fute his con­clu­sion.” – (source)

“…Neu­mark and Wascher, a study which Eliz­a­beth War­ren de­bunked in a Se­nate hear­ing” (source)

So if you’re con­ser­va­tive, Neu­mark and Wascher de­bunked Card and Krueger. But if you’re liberal, Card and Krueger de­bunked Neu­mark and Wascher.

Both sides are no doubt very pleased with them­selves. They’re not men of one study. They look at all of the re­search – ex­cept of course the stud­ies that have been “de­bunked” or “well-re­futed”. Why would you waste your time with those?

IV.

Once again, I’m not preach­ing rad­i­cal skep­ti­cism.

First of all, some stud­ies are su­per-de­bunked. Wakefield is a good ex­am­ple.

Se­cond of all, some stud­ies that don’t quite meet Wakefield-level of awful­ness are in­deed re­ally bad and need re­fut­ing. I don’t think this is be­yond the in­tel­lec­tual ca­pac­i­ties of most peo­ple. I think in many cases it’s easy to un­der­stand why a study is wrong, you should try to do that, and once you do it you can safely dis­count the re­sults of the study.

I’m not against point­ing out when you dis­agree with stud­ies or think they’re flawed. I’d be a gi­ant hyp­ocrite if I was.

But “de­bunked” and “re­futed” aren’t say­ing you dis­agree with a study. They’re mak­ing ar­gu­ments from au­thor­ity. They’re say­ing “the au­thor­ity of the sci­en­tific com­mu­nity has come to­gether and said this is a piece of crap that doesn’t count”.

And that’s fine if that’s ac­tu­ally hap­pened. But you had bet­ter make sure that you’re call­ing upon an ex cathe­dra state­ment by the com­mu­nity it­self, and not a sin­gle guy with an axe to grind. Or one side of a com­pli­cated an in­ter­minable de­bate where both sides have about equal cre­den­tials and sway.

If you can’t do that, you say “I think that my side of the aca­demic de­bate is in the right, and here’s why,” not “your side has been de­bunked”.

Other­wise you’re go­ing to end up like the min­i­mum wage de­baters, where both sides claim to have de­bunked the other. Or like the Fed­er­al­ist ar­ti­cle that says a study has been “put to bed” as “bo­gus” just be­cause an­other study said some­thing differ­ent.

I think this is part of my re­ply to the claim that em­piri­cism is so great that no one needs ra­tio­nal­ity.

A naive em­piri­cist who swears off crit­i­cal think­ing be­cause they can just “fol­low the ev­i­dence” has no con­tin­gency plan for when the ev­i­dence gets con­fus­ing. Their only re­course is to deny that the ev­i­dence is con­fus­ing, to as­sert that one side or the other has been “de­bunked”. Since they’ve already made a prin­ci­pled de­ci­sion not to study con­fir­ma­tion bias, chances are it’s go­ing to be whichever side they don’t like that’s “already been de­bunked”. And by “de­bunked” they mean “a sci­en­tist on my side said it was wrong, so now I am re­lieved from the bur­den of think­ing about it.”

On the origi­nal post, I wrote:

Life is made up of limited, con­fus­ing, con­tra­dic­tory, and mal­i­ciously doc­tored facts. Any­one who says oth­er­wise is ei­ther stick­ing to such in­cred­ibly easy solved prob­lems that they never en­counter any­thing out­side their com­fort level, or so closed-minded that they shut out any ev­i­dence that challenges their be­liefs.

In the ab­sence of any ac­tual de­bunk­ing more damn­ing than a coun­ter­ar­gu­ment, “that’s been de­bunked” is the way “shuts out any ev­i­dence that challenges their be­liefs” feels from the in­side.

V.

Some­body’s go­ing to want to know what’s up with the origi­nal rape stud­ies. The an­swer is that a small part of the dis­crep­ancy is re­sponse bias on the CSAS, but most of it is that the two sur­veys en­courage re­spon­dents to define “sex­ual as­sault” in very differ­ent ways. Vox has an ex­cel­lent ar­ti­cle on this which for once I 100% en­dorse.

In other words, both are valid, both come to­gether to form a more nu­anced pic­ture of cam­pus vi­o­lence, and nei­ther one “de­bunks” the other.