Sufficiently sincere confirmation bias is indistinguishable from science

Some the­ater peo­ple at NYU peo­ple wanted to demon­strate how gen­der stereo­types af­fected the 2016 US pres­i­den­tial elec­tion. So they de­cided to put on a the­atri­cal perfor­mance of the pres­i­den­tial de­bates – but with the gen­ders of the prin­ci­pals swapped. They as­sumed that this would show how much of a dis­ad­van­tage Hillary Clin­ton was work­ing un­der be­cause of her gen­der. They were shocked to dis­cover the op­po­site – au­di­ences full of Clin­ton sup­port­ers, watch­ing the gen­der-swapped de­bates, came away think­ing that Trump was a bet­ter com­mu­ni­ca­tor than they’d thought.

The prin­ci­pals don’t seem to have come into this with a fair-minded at­ti­tude. In­stead, it seems to have been a case of “I’ll show them!”:

Sal­va­tore says he and Guadalupe be­gan the pro­ject as­sum­ing that the gen­der in­ver­sion would con­firm what they’d each sus­pected watch­ing the real-life de­bates: that Trump’s ag­gres­sion—his ten­dency to in­ter­rupt and at­tack—would never be tol­er­ated in a woman, and that Clin­ton’s com­pe­tence and pre­pared­ness would seem even more con­vinc­ing com­ing from a man.

Let’s be clear about this. This was not epistemic even-hand­ed­ness. This was a sincere at­tempt at con­fir­ma­tion bias. They be­lieved one thing, and looked only for con­firm­ing ev­i­dence to prove their point. It was only when they started ac­tu­ally putting to­gether the ex­per­i­ment that they re­al­ized they might learn the op­po­site les­son:

But the les­sons about gen­der that emerged in re­hearsal turned out to be much less tidy. What was Jonathan Gor­don smil­ing about all the time? And didn’t he seem a lit­tle stiff, teth­ered to re­hearsed state­ments at the podium, while Brenda King, plain­spo­ken and con­fi­dent, freely roamed the stage? Which one would au­di­ences find more like­able?

What made this work? I think what hap­pened is that they took their own be­liefs liter­ally. They ac­tu­ally be­lieved that peo­ple hated Hillary be­cause she was a woman, and so their idea of some­thing that they were con­fi­dent would show this clearly was a fair test. Be­cause of this, when things came out the op­po­site of the way they’d pre­dicted, they no­ticed and were sur­prised, be­cause they ac­tu­ally ex­pected the demon­stra­tion to work.

But they went fur­ther. Even though they knew in ad­vance of the pub­lic perfor­mances that the ex­per­i­ment got the wrong an­swer, they nei­ther falsified nor file-draw­ered the ev­i­dence. They tried to show, they got a differ­ent an­swer, they showed it any­way.

This is much, much bet­ter sci­ence than con­tem­po­rary med­i­cal or psy­chol­ogy re­search were be­fore the repli­ca­tion crisis.

Some­times, when I think about how epistem­i­cally cor­rupt our cul­ture is, I’m tempted to adopt a per­ma­nent defen­sive crouch and dis­be­lieve any­thing I can’t fact-check, to ex­plic­itly ad­just for all the rele­vant bi­ases, and this prospect sounds ex­haust­ing. It’s not ac­tu­ally nec­es­sary. You don’t have to worry too much about your bi­ases. Just take your own be­liefs liter­ally, as though they mean what they say they mean, and try to be­lieve all their con­se­quences as well. And, when you hit a con­tra­dic­tion – well, now you have an op­por­tu­nity to learn where you’re wrong.

(Cross-posted at my per­sonal blog.)