“No evidence” as a Valley of Bad Rationality

Quick sum­mary of Doc­tor, There are Two Kinds of “No Ev­i­dence”:

  • Author has a rel­a­tive with can­cer. Rel­a­tive is do­ing well af­ter chemo and is go­ing to a doc­tor to see if it’s worth get­ting more chemo to kill the lit­tle ex­tra bits of can­cer that might be lin­ger­ing.

  • Doc­tor says that there is no ev­i­dence that get­ting more chemo does any good in these situ­a­tions.

  • Author says that this vi­o­lates com­mon sense.

  • Doc­tor says that com­mon sense doesn’t mat­ter, ev­i­dence does.

  • Author asks whether “no ev­i­dence” means 1) a lot of stud­ies show­ing that it doesn’t do any good, or 2) not enough stud­ies to con­clu­sively say that it does good.

  • Doc­tor didn’t un­der­stand the differ­ence.

Let me be clear about the mis­take the doc­tor is mak­ing: he’s fo­cused on con­clu­sive ev­i­dence. To him, if the ev­i­dence isn’t con­clu­sive, it doesn’t count.

I think this doc­tor is stuck in a Valley of Bad Ra­tion­al­ity. Here’s what I mean:

  • The av­er­age Joe doesn’t know any­thing about t-tests and p-val­ues, but the av­er­age Joe does know to up­date his be­liefs in­cre­men­tally. La­mar Jack­son just had an­other 4 touch­down game? It’s not con­clu­sive, but it starts to point more in the di­rec­tion of him win­ning the MVP.

  • The av­er­age Joe doesn’t know any­thing about for­mal statis­ti­cal meth­ods. He up­dates his be­liefs in a hand-wavy, wishy-washy way.

  • The doc­tor went to school to learn about these for­mal statis­ti­cal meth­ods. He learned that the­o­riz­ing is er­ror prone and that we need to base our be­liefs on hard data. And he learned that if our p-value isn’t less than 0.05, we can’t re­ject the null hy­poth­e­sis.

  • You can ar­gue that so far, the doc­tors ed­u­ca­tion didn’t move him for­ward. That it in­stead caused him to take a step back­wards. Think about it: he’s tel­ling a pa­tient with can­cer to not even con­sider more chemo be­cause there is “no ev­i­dence” that it will do “any” good. I think Aver­age Joe could do bet­ter than that.

  • But if the doc­tor con­tinued his ed­u­ca­tion and learned more about statis­tics, he’d learn that his in­tro class didn’t paint a com­plete pic­ture. He’d learn that you don’t always have ac­cess to “con­clu­sive” ev­i­dence, and that in these situ­a­tions, some­times you just have to work with what you have. He’d also learn that he was priv­ileg­ing the null hy­poth­e­sis in a situ­a­tion where it’d make sense to do the op­po­site. The null hy­poth­e­sis of “more chemo has no effect” prob­a­bly isn’t true.

  • Once the doc­tor re­ceives this fur­ther ed­u­ca­tion, it’d push him two steps for­ward.

  • In the in­tro class, he took one step back­wards. At that point he’s in the Valley of Bad Ra­tion­al­ity: ed­u­ca­tion made him worse than where he started. But then when he re­ceived more ed­u­ca­tion, he took two steps for­ward. It brought him out of this valley and fur­ther along than where he started.

I think that a lot of peo­ple are stuck in this same valley.