Schools Proliferating Without Evidence

Robyn Dawes, au­thor of one of the origi­nal pa­pers from Judg­ment Un­der Uncer­tainty and of the book Ra­tional Choice in an Uncer­tain World—one of the few who tries re­ally hard to im­port the re­sults to real life—is also the au­thor of House of Cards: Psy­chol­ogy and Psy­chother­apy Built on Myth.

From House of Cards, chap­ter 1:

The abil­ity of these pro­fes­sion­als has been sub­jected to em­piri­cal scrutiny—for ex­am­ple, their effec­tive­ness as ther­a­pists (Chap­ter 2), their in­sight about peo­ple (Chap­ter 3), and the re­la­tion­ship be­tween how well they func­tion and the amount of ex­pe­rience they have had in their field (Chap­ter 4). Vir­tu­ally all the re­search—and this book will refer­ence more than three hun­dred em­piri­cal in­ves­ti­ga­tions and sum­maries of in­ves­ti­ga­tions—has found that these pro­fes­sion­als’ claims to su­pe­rior in­tu­itive in­sight, un­der­stand­ing, and skill as ther­a­pists are sim­ply in­valid...

Re­mem­ber Rorschach ink-blot tests? It’s such an ap­peal­ing ar­gu­ment: the pa­tient looks at the ink-blot and says what he sees, the psy­chother­a­pist in­ter­prets their psy­cholog­i­cal state based on this. There’ve been hun­dreds of ex­per­i­ments look­ing for some ev­i­dence that it ac­tu­ally works. Since you’re read­ing this, you can guess the an­swer is sim­ply “No.” Yet the Rorschach is still in use. It’s just such a good story that psy­chother­a­pists just can’t bring them­selves to be­lieve the vast mounds of ex­per­i­men­tal ev­i­dence say­ing it doesn’t work—

—which tells you what sort of field we’re deal­ing with here.

And the ex­per­i­men­tal re­sults on the field as a whole are com­men­su­rate. Yes, pa­tients who see psy­chother­a­pists have been known to get bet­ter faster than pa­tients who sim­ply do noth­ing. But there is no statis­ti­cally dis­cernible differ­ence be­tween the many schools of psy­chother­apy. There is no dis­cernible gain from years of ex­per­tise.

And there’s also no dis­cernible differ­ence be­tween see­ing a psy­chother­a­pist and spend­ing the same amount of time talk­ing to a ran­domly se­lected col­lege pro­fes­sor from an­other field. It’s just talk­ing to any­one that helps you get bet­ter, ap­par­ently.

In the en­tire ab­sence of the slight­est ex­per­i­men­tal ev­i­dence for their effec­tive­ness, psy­chother­a­pists be­came li­censed by states, their tes­ti­mony ac­cepted in court, their teach­ing schools ac­cred­ited, and their bills paid by health in­surance.

And there was also a huge pro­lifer­a­tion of “schools”, of tra­di­tions of prac­tice, in psy­chother­apy; de­spite—or per­haps be­cause of—the lack of any ex­per­i­ments show­ing that one school was bet­ter than an­other...

I should re­ally post more some other time on all the sad things this says about our world; about how the essence of medicine, as rec­og­nized by so­ciety and the courts, is not a reper­toire of pro­ce­dures with statis­ti­cal ev­i­dence for their heal­ing effec­tive­ness; but, rather, the right air of au­thor­ity.

But the sub­ject to­day is the pro­lifer­a­tion of tra­di­tions in psy­chother­apy. So far as I can dis­cern, this was the way you picked up pres­tige in the field—not by dis­cov­er­ing an amaz­ing new tech­nique whose effec­tive­ness could be ex­per­i­men­tally ver­ified and adopted by all; but, rather, by split­ting off your own “school”, sup­ported by your charisma as founder, and by the good sto­ries you told about all the rea­sons your tech­niques should work.

This was prob­a­bly, to no small ex­tent, re­spon­si­ble for the ex­is­tence and con­tinu­a­tion of psy­chother­apy in the first place—the promise of mak­ing your­self a Master, like Freud who’d done it first (also with­out the slight­est scrap of ex­per­i­men­tal ev­i­dence). That’s the brass ring of suc­cess to chase—the prospect of be­ing a guru and hav­ing your own ad­her­ents. It’s the strug­gle for ad­her­ents that keeps the clergy vi­tal.

That’s what hap­pens to a field when it un­binds it­self from the ex­per­i­men­tal ev­i­dence—though there were other fac­tors that also placed psy­chother­a­pists at risk, such as the defer­ence shown them by their pa­tients, the wish of so­ciety to be­lieve that men­tal heal­ing was pos­si­ble, and, of course, the gen­eral dan­gers of tel­ling peo­ple how to think.

The field of he­do­nic psy­chol­ogy (hap­piness stud­ies) be­gan, to some ex­tent, with the re­al­iza­tion that you could mea­sure hap­piness—that there was a fam­ily of mea­sures that by golly did val­i­date well against each other.

The act of cre­at­ing a new mea­sure­ment cre­ates new sci­ence; if it’s a good mea­sure­ment, you get good sci­ence.

If you’re go­ing to cre­ate an or­ga­nized prac­tice of any­thing, you re­ally do need some way of tel­ling how well you’re do­ing, and a prac­tice of do­ing se­ri­ous test­ing—that means a con­trol group, an ex­per­i­men­tal group, and statis­tics—on plau­si­ble-sound­ing tech­niques that peo­ple come up with. You re­ally need it.

Added: Dawes wrote in the 80s and I know that the Rorschach was still in use as re­cently as the 90s, but it’s pos­si­ble mat­ters have im­proved since then (as one com­menter states). I do re­mem­ber hear­ing that there was pos­i­tive ev­i­dence for the greater effec­tive­ness of cog­ni­tive-be­hav­ioral ther­apy.