MetaMed: Evidence-Based Healthcare

In a world where 85% of doc­tors can’t solve sim­ple Bayesian word prob­lems...

In a world where only 20.9% of re­ported re­sults that a phar­ma­ceu­ti­cal com­pany tries to in­ves­ti­gate for de­vel­op­ment pur­poses, fully repli­cate...

In a world where “p-val­ues” are any­thing the au­thor wants them to be...

...and where there are all sorts of amaz­ing tech­nolo­gies and tech­niques which no­body at your hos­pi­tal has ever heard of...

...there’s also Me­taMed. In­stead of just hav­ing “ev­i­dence-based medicine” in jour­nals that doc­tors don’t ac­tu­ally read, Me­taMed will provide you with ac­tual ev­i­dence-based health­care. Their Chair­man and CTO is Jaan Tal­linn (cofounder of Skype, ma­jor fun­der of xrisk-re­lated en­deav­ors), one of their ma­jor VCs is Peter Thiel (ma­jor fun­der of MIRI), their man­age­ment in­cludes some names LWers will find fa­mil­iar, and their re­searchers know math and stats and in many cases have also read LessWrong. If you have a suffi­ciently se­ri­ous prob­lem and can af­ford their ser­vice, Me­taMed will (a) put some­one on read­ing the rele­vant re­search liter­a­ture who un­der­stands real statis­tics and can tell whether the pa­per is trust­wor­thy; and (b) re­fer you to a co­op­er­a­tive doc­tor in their net­work who can carry out the ther­a­pies they find.

Me­taMed was par­tially in­spired by the case of a woman who had her finger­tip chopped off, was told by the hos­pi­tal that she was screwed, and then read through an awful lot of liter­a­ture on her own un­til she found some­one work­ing on an ad­vanced re­gen­er­a­tive ther­apy that let her ac­tu­ally grow the finger­tip back. The idea be­hind Me­taMed isn’t just that they will scour the liter­a­ture to find how the best ex­per­i­men­tally sup­ported treat­ment differs from the av­er­age wis­dom—peo­ple who reg­u­larly read LW will be aware that this is of­ten a pretty large di­ver­gence—but that they will also look for this sort of very re­cent tech­nol­ogy that most hos­pi­tals won’t have heard about.

This is a new ser­vice and it has to in­ter­act with the ex­ist­ing med­i­cal sys­tem, so they are cur­rently ex­pen­sive, start­ing at $5,000 for a re­search re­port. (Keep­ing in mind that a ba­sic re­port in­volves a lot of work by peo­ple who must be good at math.) If you have a sick friend who can af­ford it—es­pe­cially if the reg­u­lar sys­tem is failing them, and they want (or you want) their next step to be more sci­ence in­stead of “al­ter­na­tive medicine” or what­ever—please do re­fer them to Me­taMed im­me­di­ately. We can’t all have nice things like this some­day un­less some­body pays for it while it’s still new and ex­pen­sive. And the reg­u­lar health­care sys­tem re­ally is bad enough at sci­ence (es­pe­cially in the US, but sci­ence is difficult ev­ery­where) that there’s no point in con­demn­ing any­one to it when they can af­ford bet­ter.


I also got my hands on a copy of Me­taMed’s stan­dard list of cita­tions that they use to sup­port points to re­porters. What fol­lows isn’t nearly ev­ery­thing on Me­taMed’s list, just the items I found most in­ter­est­ing.


90% of pre­clini­cal can­cer stud­ies could not be repli­cated:
http://​​www.na­ture.com/​​na­ture/​​jour­nal/​​v483/​​n7391/​​full/​​483531a.html

“It is fre­quently stated that it takes an av­er­age of 17 years for re­search ev­i­dence to reach clini­cal prac­tice. Balas and Bo­hen, Grant, and Wratschko all es­ti­mated a time lag of 17 years mea­sur­ing differ­ent points of the pro­cess.”—http://​​www.jrsm.rsmjour­nals.com/​​con­tent/​​104/​​12/​​510.full
“The au­thors es­ti­mated the vol­ume of med­i­cal liter­a­ture po­ten­tially rele­vant to pri­mary care pub­lished in a month and the time re­quired for physi­ci­ans trained in med­i­cal epi­demiol­ogy to eval­u­ate it for up­dat­ing a clini­cal knowl­edge­base.… Aver­age time per ar­ti­cle was 2.89 min­utes, if this out­lier was ex­cluded. Ex­trap­o­lat­ing this es­ti­mate to 7,287 ar­ti­cles per month, this effort would re­quire 627.5 hours per month, or about 29 hours per week­day.”
One-third of hos­pi­tal pa­tients are harmed by their stay in the hos­pi­tal, and 7% of pa­tients are ei­ther per­ma­nently harmed or die: http://​​www.ama-assn.org/​​amed­news/​​2011/​​04/​​18/​​prl20418.htm
(I emailed Me­taMed to ask for the ac­tual bibliog­ra­phy for the fol­low­ing cita­tions, since that wasn’t in­cluded in the copy of the list I saw. I already rec­og­nize some of the cita­tions hav­ing to do with Bayesian rea­son­ing, which makes me fairly con­fi­dent of the oth­ers.)
Statis­ti­cal Illiteracy
Doc­tors of­ten con­fuse sen­si­tivity and speci­fic­ity (Gigeren­zer 2002); most physi­ci­ans do not un­der­stand how to com­pute the pos­i­tive pre­dic­tive value of a test (Hoffrage and Gigeren­zer 1998); a third over­es­ti­mate benefits if they are ex­pressed as pos­i­tive risk re­duc­tions (Gigeren­zer et al 2007).
Physi­ci­ans think a pro­ce­dure is more effec­tive if the benefits are de­scribed as a rel­a­tive risk re­duc­tion rather than as an ab­solute risk re­duc­tion (Nay­lor et al 1992).
Only 3 out of 140 re­view­ers of four breast can­cer screen­ing pro­pos­als no­ticed that all four were iden­ti­cal pro­pos­als with the risks rep­re­sented differ­ently (Fa­hey et al 1995).
60% of gy­ne­col­o­gists do not un­der­stand what the sen­si­tivity and speci­fic­ity of a test are (Gigeren­zer at al 2007).
95% of physi­ci­ans over­es­ti­mated the prob­a­bil­ity of breast can­cer given a pos­i­tive mam­mo­gram by an or­der of mag­ni­tude (Eddy 1982).
When physi­ci­ans re­ceive prostate can­cer screen­ing in­for­ma­tion in terms of five-year sur­vival rates, 78% think screen­ing is effec­tive; when the same in­for­ma­tion is given in terms of mor­tal­ity rates, 5% be­lieve it is effec­tive (Weg­warth et al, sub­mit­ted).
Only one out of 21 ob­ste­tri­ci­ans could es­ti­mate the prob­a­bil­ity that an un­born child had Down syn­drome given a pos­i­tive test (Bramwell, West, and Sal­mon 2006).
Six­teen out of twenty HIV coun­selors said that there was no such thing as a false pos­i­tive HIV test (Gigeren­zer et all 1998).
Only 3% of ques­tions in the cer­tifi­ca­tion exam for the Amer­i­can Board of In­ter­nal Medicine cover clini­cal epi­demiol­ogy or med­i­cal statis­tics, and risk com­mu­ni­ca­tion is not ad­dressed (Gigeren­zer et al 2007).
Bri­tish GPs rarely change their pre­scribing pat­terns and when they do it’s rarely in re­sponse to ev­i­dence (Arm­strong et al 1996).
Drug Advertising
Direct-to-cus­tomer ad­ver­tis­ing by phar­ma­ceu­ti­cal com­pa­nies, which is in­tended to sell drugs rather than to ed­u­cate, of­ten does not con­tain in­for­ma­tion about a drug’s suc­cess rate (only 9% did), al­ter­na­tive meth­ods of treat­ment (29%), be­hav­ioral changes (24%), or the treat­ment du­ra­tion (9%) (Bell et al 2000).
Pa­tients are more likely to re­quest ad­ver­tised drugs and doc­tors to pre­scribe them, re­gard­less of their mis­giv­ings (Gil­body et al 2005).
Med­i­cal Errors
44,000 to 98,000 pa­tients are kil­led in US hos­pi­tals each year by doc­u­mented, pre­ventable med­i­cal er­rors (Kohn et al 2000).
De­spite proven effec­tive­ness of sim­ple check­lists in re­duc­ing in­fec­tions in hos­pi­tals (Provonost et al 2006), most ICU physi­ci­ans do not use them.
Sim­ple di­ag­nos­tic tools which may even ig­nore some data give mea­surably bet­ter out­comes in ar­eas such as de­cid­ing whether to put a new ad­mis­sion in a coro­nary care bed (Green and Mehr 1997).
Tort law of­ten ac­tively pe­nal­izes physi­ci­ans who prac­tice ev­i­dence-based medicine in­stead of the medicine that is cus­tom­ary in their area (Mon­a­han 2007).
Out of 175 law schools, only one re­quires a ba­sic course in statis­tics or re­search meth­ods (Faig­man 1999), so many judges, ju­rors, and lawyers are mis­led by non­trans­par­ent statis­tics.
93% of sur­geons, ob­stret­i­ci­ans, and other health care pro­fes­sion­als at high risk for malprac­tice suits re­port prac­tic­ing defen­sive medicine (Stud­dert et al 2005).
Re­gional Vari­a­tions in Health Care
Ton­sillec­tomies vary twelve­fold be­tween the coun­ties in Ver­mont with the high­est and low­est rates of the pro­ce­dure (Wennberg and Git­tel­sohn 1973).
Five­fold vari­a­tions in one-year sur­vival from can­cer across differ­ent re­gions have been ob­served (Quam and Smith 2005).
Fif­tyfold vari­a­tions in peo­ple re­ceiv­ing drug treat­ment for de­men­tia has been re­ported (Pre­scribing Ob­ser­va­tory for Men­tal Health 2007).
Rates of cer­tain sur­gi­cal pro­ce­dures vary ten­fold to fif­teen­fold be­tween re­gions (McPher­son et al 1982).
Clini­ci­ans are more likely to con­sult their col­leagues than med­i­cal jour­nals or the library, par­tially ex­plain­ing re­gional differ­ences (Shaugh­nessy et al 1994).
Research
Re­searchers may re­port only fa­vor­able tri­als, only re­port fa­vor­able data (An­gell 2004), or cherry-pick data to only re­port fa­vor­able vari­ables or sub­groups (Ren­nie 1997).
Of 50 sys­tem­atic re­views and meta-analy­ses on asthma treat­ment 40 had se­ri­ous or ex­ten­sive flaws, in­clud­ing all 6 as­so­ci­ated with in­dus­try (Jadad et al 2000).
Less high-tech knowl­edge and ap­pli­ca­tions tend to be con­sid­ered less in­no­va­tive and ig­nored (Shi and Singh 2008).
Poor Use of Statis­tics In Research
Only about 7% of ma­jor-jour­nal tri­als re­port re­sults us­ing trans­par­ent statis­tics (Nuovo, Melnivov and Chang 2002).
Data are of­ten re­ported in bi­ased ways: for in­stance, benefits are of­ten re­ported as rel­a­tive risks (“re­duces the risk by half”) and harms as ab­solute risks (“an in­crease of 5 in 1000”); ab­solute risks seem smaller even when the risk is the same (Gigeren­zer et al 2007).
Half of tri­als in­ap­pro­pri­ately use sig­nifi­cance tests for baseline com­par­i­son; 23 pre­sent sub­group find­ings, a sign of pos­si­ble data fish­ing, of­ten with­out ap­pro­pri­ate tests for in­ter­ac­tion (Ass­man et al 2000).
One third of stud­ies use mis­matched fram­ing, where benefits are re­ported one way (usu­ally rel­a­tive risk re­duc­tion, which makes them look big­ger) and harms an­other (usu­ally ab­solute risk re­duc­tion, which makes them look smaller) (Se­drakyan and Shih 2007).
Pos­i­tive Publi­ca­tion Bias
Pos­i­tive pub­li­ca­tion bias over­states the effects of treat­ment by up to one-third (Schultz et al 1995).
More than 50% of re­search is un­pub­lished or un­re­ported (Mathieu et al 2009).
In ten high-im­pact med­i­cal jour­nals, only 45.5% of tri­als were ad­e­quately reg­istered be­fore test­ing be­gan; of these 31% show dis­crep­an­cies be­tween out­comes mea­sured and pub­lished (Mathieu et al 2009).
Phar­ma­ceu­ti­cal Com­pany In­duced Bias
Stud­ies funded by the phar­ma­ceu­ti­cal in­dus­try are more likely to re­port re­sults fa­vor­able to the spon­sor­ing com­pany (Lexchin et al 2003).
There is a sig­nifi­cant as­so­ci­a­tion be­tween in­dus­try spon­sor­ship and both pro-in­dus­try out­comes and poor method­ol­ogy (Bekel­man and Kron­mal 2008).
In man­u­fac­turer-sup­ported tri­als of non-steroidal anti-in­flam­ma­tory drugs, half the time the data pre­sented did not match claims made within the ar­ti­cle (Ro­chon et al 1994).
68% of US health re­search is funded by in­dus­try (Re­search!Amer­ica 2008), which means that re­search that leads to prof­its to the health care in­dus­try tends to be pri­ori­tized.
71 out of 78 drugs ap­proved by the FDA in 2002 are “me too” drugs that are more prof­itable be­cause of the patent but not sub­stan­tially differ­ent from ex­ist­ing med­i­ca­tion (An­gell 2004).
“Seed­ing tri­als” by phar­ma­ceu­ti­cal com­pa­nies pro­mote treat­ments in­stead of test­ing hy­pothe­ses (Hill et al 2008).
Even ac­cu­rate re­search may be mis­re­ported by phar­ma­ceu­ti­cal com­pany ad­ver­tis­ing, in­clud­ing ads in med­i­cal jour­nals (Villanueva et al 2003).
In 92% of cases, phar­ma­ceu­ti­cal leaflets dis­tributed to doc­tors have data sum­maries that ei­ther can­not be ver­ified or in­ac­cu­rately sum­ma­rize available data (Kaiser et al 2004).
I don’t plan on be­com­ing se­ri­ously sick, but if I do, I think I’ll check in with Me­taMed just to make sure no­body is ig­nor­ing the re­search re­sults show­ing that you shouldn’t feed the pa­tient rat poi­son.