Moloch’s Toolbox (1/​2)

Fol­low-up to: An Equil­ibrium of No Free Energy


There’s a toolbox of reusable con­cepts for an­a­lyz­ing sys­tems I would call “in­ad­e­quate”—the causes of civ­i­liza­tional failure, some of which cor­re­spond to lo­cal op­por­tu­ni­ties to do bet­ter your­self. I shall, some­what ar­bi­trar­ily, sort these con­cepts into three larger cat­e­gories:

1. De­ci­sion­mak­ers who are not benefi­cia­ries;

2. Asym­met­ric in­for­ma­tion;

and above all,

3. Nash equil­ibria that aren’t even the best Nash equil­ibrium, let alone Pareto-op­ti­mal.

In other words:

1. Cases where the de­ci­sion lies in the hands of peo­ple who would gain lit­tle per­son­ally, or lose out per­son­ally, if they did what was nec­es­sary to help some­one else;

2. Cases where de­ci­sion-mak­ers can’t re­li­ably learn the in­for­ma­tion they need to make de­ci­sions, even though some­one else has that in­for­ma­tion; and

3. Sys­tems that are bro­ken in mul­ti­ple places so that no one ac­tor can make them bet­ter, even though, in prin­ci­ple, some mag­i­cally co­or­di­nated ac­tion could move to a new sta­ble state.

I will then play fast and loose with these con­cepts in or­der to fit the en­tire Tax­on­omy of Failure in­side them.

For ex­am­ple, “ir­ra­tional­ity in the form of cog­ni­tive bi­ases” wouldn’t ob­vi­ously fit into any of these cat­e­gories, but I’m go­ing to shove it in­side “asym­met­ric in­for­ma­tion” via a clever sleight-of-hand. Ready? Here goes:

If no­body can de­tect a cog­ni­tive bias in par­tic­u­lar cases, then from our per­spec­tive we can’t re­ally call it a “civ­i­liza­tional in­ad­e­quacy” or “failure to pluck a low-hang­ing fruit.” We shouldn’t even be able to see it our­selves. So, on the con­trary, let’s sup­pose that you and some other peo­ple can in­deed de­tect a cog­ni­tive bias that’s screw­ing up civ­i­liza­tional de­ci­sion­mak­ing.

Then why don’t you just walk up to the de­ci­sion-maker and tell them about the bias? Be­cause they wouldn’t have any way of know­ing to trust you rather than the other five hun­dred peo­ple try­ing to in­fluence their de­ci­sions? Well, in that case, you’re hold­ing in­for­ma­tion that they can’t learn from you! So that’s an “asym­met­ric in­for­ma­tion prob­lem,” in much the same way that it’s an asym­met­ric in­for­ma­tion prob­lem when you’re try­ing to sell a used car and you know it doesn’t have any me­chan­i­cal prob­lems, but you have no way of re­li­ably con­vey­ing this knowl­edge to the buyer be­cause for all they know you could be ly­ing.

That ar­gu­ment is a bit silly, but so is the no­tion of try­ing to fit the whole Scroll of Woe into three su­per­cat­e­gories. And if I named more than three su­per­cat­e­gories, you wouldn’t be able to re­mem­ber them due to com­pu­ta­tional limi­ta­tions (which aren’t on the list any­where, and I’m not go­ing to add them).

i. For want of do­cosa­hex­aenoic acids, a baby was lost

My dis­cus­sion of mod­est episte­mol­ogy in Chap­ter 1 might have given the im­pres­sion that I think of mod­esty mostly as a cer­tain set of high-level be­liefs: be­liefs about how best to com­bat cog­ni­tive bias, about how in­di­vi­d­ual com­pe­ten­cies stack up against group-level com­pe­ten­cies, and so on. But I pre­dict that many of this book’s read­ers have high-level be­liefs similar to those I out­lined in Chap­ter 2, while em­ploy­ing a rea­son­ing style that is re­ally a spe­cial case of mod­est episte­mol­ogy; and I think that this rea­son­ing style is caus­ing them sub­stan­tial harm.

As rea­son­ing styles, mod­est episte­mol­ogy and in­ad­e­quacy anal­y­sis de­pend on a mix of ex­plicit prin­ci­ples and im­plicit men­tal habits. In in­ad­e­quacy anal­y­sis, it’s one thing to rec­og­nize in the ab­stract that we live in a world rife with sys­temic in­effi­cien­cies, and quite an­other to nat­u­rally per­ceive sys­tems that way in daily life. So my goal here won’t be to un­kindly stick the la­bel “in­ad­e­quate” to a black box con­tain­ing the world; it will be to say some­thing about how the rele­vant sys­tems ac­tu­ally op­er­ate.

For our cen­tral ex­am­ple, we’ll be us­ing the United States med­i­cal sys­tem, which is, so far as I know, the most bro­ken sys­tem that still works ever recorded in hu­man his­tory. If you were read­ing about some­thing in 19th-cen­tury France which was as bro­ken as US health­care, you wouldn’t ex­pect to find that it went on work­ing when over­loaded with a suffi­ciently vast amount of money. You would ex­pect it to just not work at all.

In pre­vi­ous years, I would use the case of cen­tral-line in­fec­tions as my go-to ex­am­ple of med­i­cal in­ad­e­quacy. Cen­tral-line in­fec­tions, in the US alone, kil­led 60,000 pa­tients per year, and in­fected an ad­di­tional 200,000 pa­tients at an av­er­age treat­ment cost of $50,000/​pa­tient.

Cen­tral-line in­fec­tions were also known to de­crease by 50% or more if you en­forced a five-item check­list that in­cluded items like “wash your hands be­fore touch­ing the line.”

Robin Han­son has old Over­com­ing Bias blog posts on that un­taken, low-hang­ing fruit. But I dis­cov­ered while re-Googling in 2015 that wider adop­tion of hand-wash­ing and similar pre­cau­tions are now fi­nally be­gin­ning to oc­cur, af­ter many years—with an as­so­ci­ated 43% na­tion­wide de­crease in cen­tral-line in­fec­tions. After par­tial adop­tion.1

So my new ex­am­ple is in­fants suffer­ing liver dam­age, brain dam­age, and death in a way that’s even eas­ier to solve, by chang­ing the lipid dis­tri­bu­tion of par­enteral nu­tri­tion to match the pro­por­tions in breast milk.

Back­ground: Some ba­bies have di­ges­tion prob­lems that re­quire di­rect in­tra­venous feed­ing. Long ago, some­body cre­ated a hos­pi­tal for­mula for this in­tra­venous feed­ing that matched the dis­tri­bu­tion of “fat,” “pro­tein,” and “car­bo­hy­drate” in breast milk.

Just like “pro­tein” comes in differ­ent amino acids, some of which the body can’t make on its own and some of which it can, what early doc­tors used to think of as “fat” ac­tu­ally breaks down into metabol­i­cally dis­tinct el­e­ments like short-chain triglyc­erides, medium-chain triglyc­erides, sat­u­rated fat, and omega-6, omega-9, and the fa­mous “omega-3.” “Omega-3” is ac­tu­ally sev­eral differ­ent lipids in its own right; veg­etable oils with “omega-3” usu­ally just con­tain alpha-linolenic acids, which can only be in­effi­ciently con­verted to ecos­apen­taenoic acids, which are then even more in­effi­ciently con­verted to do­cosa­hex­aenoic acids, which are the ac­tual key struc­tural com­po­nents in the body. This con­ver­sion path­way is rate-limited by a pro­cess that also con­verts omega-6, so too much omega-6 can pre­vent you from pro­cess­ing ALA into DHA even if you’re get­ting ALA.

So what hap­pens if your in­fant nu­tri­tion was ini­tially de­signed based on the con­cept of “fat” as a nat­u­ral cat­e­gory, and all the “fat” in the mix comes from soy­bean oil?

From a pop­u­lar book by Jaminet and Jaminet:

Some ba­bies are born with “short bowel syn­drome” and need to be given par­enteral nu­tri­tion, or nu­tri­tion de­liv­ered in­tra­venously di­rectly to the blood, un­til their di­ges­tive tracts grow and heal. Since 1961, par­enteral nu­tri­tion has used soy­bean oil as its source of fat.[6] And for decades, ba­bies on par­enteral nu­tri­tion have suffered dev­as­tat­ing liver and brain dam­age. The death rate on soy­bean oil is 30 per­cent by age four. […]

In a clini­cal trial, of forty-two ba­bies given fish oil [af­ter they had already de­vel­oped liver dam­age on soy­bean oil], three died and one re­quired a liver trans­plant; of forty-nine given soy­bean oil, twelve died and six re­quired a liver trans­plant.[8] The death-or-liver-trans­plant rate was re­duced from 37 per­cent with soy­bean oil to 9 per­cent with fish oil.2

When Jaminet and Jaminet wrote the above, in 2012, there was a sin­gle hos­pi­tal in the United States that could provide cor­rectly for­mu­lated par­enteral nu­tri­tion, namely the Bos­ton Chil­dren’s Hospi­tal; nowhere else. This for­mu­la­tion was ille­gal to sell across state lines.

A few years af­ter the Bos­ton Chil­dren’s Hospi­tal de­vel­oped their for­mula—keep­ing in mind the heap of dead ba­bies con­tin­u­ing to pile up in the mean­while—there de­vel­oped a short­age of “cer­tified lipids” (FDA-ap­proved “fat” for adding to par­enteral nu­tri­tion). For a year or two, the par­enteral nu­tri­tion con­tained no fat at all which is worse and can kill adults.

You see, al­though there’s noth­ing spe­cial about the soy­bean oil in par­enteral nu­tri­tion, there was only one US man­u­fac­turer ap­proved to add it, and that man­u­fac­turer left the mar­ket, so…

As of 2015, the state of af­fairs was as fol­lows: The FDA even­tu­ally solved the prob­lem with the short­age of US-cer­tified lipids, by… al­low­ing US hos­pi­tals to im­port par­enteral nu­tri­tion bags from Europe. And it only took them two years’ worth of dead pa­tients to figure that out!

As of 2016, if your baby has short bowel syn­drome, and has already ended up with liver dam­age, and ei­ther you or your doc­tor is lucky enough to know what’s wrong and how to fix it, your doc­tor can ap­ply for a spe­cial per­mit to use a non-FDA-ap­proved sub­stance for your child on an emer­gency ba­sis. After this, you can buy Omegaven and hope that it cures your baby and that there isn’t too much per­ma­nent dam­age and that it’s not already too late.

This is an im­prove­ment over the prior situ­a­tion, where the non-poi­sonous for­mu­la­tion was ille­gal to sell across state lines un­der any cir­cum­stances, but it’s still not good by any stretch of the imag­i­na­tion.

Now imag­ine try­ing to ex­plain to a vis­i­tor from a rel­a­tively well-func­tion­ing world just why it is that your civ­i­liza­tion has kil­led a bunch of ba­bies and sub­jected other ba­bies to pointless brain dam­age.

“It’s not that we’re evil,” you say hel­plessly, “it’s that… well, you see, it’s not that any­one wanted to kill those ba­bies, it’s just the way the Sys­tem ended up, some­how…”

ii. Asym­met­ric in­for­ma­tion and lemons problems

Three peo­ple have gath­ered in a blank white space:

  • The vis­i­tor from a Bet­ter World;

  • sim­pli­cio , who is at­tend­ing a ma­jor uni­ver­sity but hasn’t taken un­der­grad­u­ate eco­nomics;
  • ce­cie, the Con­ven­tional Cyn­i­cal Economist.

The Visi­tor speaks first.

vis­i­tor: So I’ve listened to you ex­plain about ba­bies suffer­ing death and brain dam­age from par­enteral nu­tri­tion built on soy­bean oil. I have sev­eral ques­tions here, but I’ll start with the most ob­vi­ous one.

ce­cie: Go ahead.

vis­i­tor: Why aren’t there ri­ots?

sim­pli­cio: The first thing you have to un­der­stand, Visi­tor, is that the folk in this world are hyp­ocrites, cow­ards, psy­chopaths, and sheep.

I mean, I cer­tainly care about the the lives of new­born chil­dren. Hear­ing about their plight cer­tainly makes me want to do some­thing about it. When I see the prob­lem con­tin­u­ing in spite of that, I can only con­clude that other peo­ple don’t feel the level of moral in­dig­na­tion that I feel when star­ing at a heap of dead ba­bies.

ce­cie: I don’t think that hy­poth­e­sis is needed, Sim­pli­cio. As a start, Visi­tor, you have to re­al­ize that the pic­ture I’ve shown you is not widely known. Maybe 10% of the pop­u­la­tion, at most, is walk­ing around with the prior be­lief that the FDA in gen­eral is kil­ling peo­ple; our gov­ern­ment runs on ma­jor­ity rule and the 10% can’t unilat­er­ally defy it.3 Maybe 0.1% of that 10% know that omega-3 ALA is con­verted into omega-3 DHA via a metabolic path­way that com­petes with omega-6. And then most of those aren’t aware of what’s hap­pen­ing to ba­bies right now.

vis­i­tor: Point­ing to that state of ig­no­rance is hardly a suffi­cient ex­pla­na­tion! If a the­ater is on fire and only one per­son knows it, they yell “Fire!” and then more peo­ple know it. Peo­ple from my civ­i­liza­tion would scream “Ba­bies are dy­ing over here!” and other peo­ple from my civ­i­liza­tion would whip around their heads and look.

sim­pli­cio: Our world’s cow­ards and sheep would hear that and think that it’s (a) some­body else’s prob­lem and (b) all part of the plan.

ce­cie: In our world, Visi­tor, we have an eco­nomic phe­nomenon some­times called the lemons prob­lem. Sup­pose you want to sell a used car, and I’m look­ing for a car to buy. From my per­spec­tive, I have to worry that your car might be a “lemon”—that it has a se­ri­ous me­chan­i­cal prob­lem that doesn’t ap­pear ev­ery time you start the car, and is difficult or im­pos­si­ble to fix. Now, you know that your car isn’t a lemon. But if I ask you, “Hey, is this car a lemon?” and you an­swer “No,” I can’t trust your an­swer, be­cause you’re in­cen­tivized to an­swer “No” ei­ther way. Hear­ing you say “No” isn’t much Bayesian ev­i­dence. Asym­met­ric in­for­ma­tion con­di­tions can per­sist even in cases where, like an hon­est sel­ler meet­ing an hon­est buyer, both par­ties have strong in­cen­tives for ac­cu­rate in­for­ma­tion to be con­veyed.

A fur­ther prob­lem is that if the fair value of a non-lemon car is $10,000, and the pos­si­bil­ity that your car is a lemon causes me to only be will­ing to pay you $8,000, you might re­fuse to sell your car. So the hon­est sel­l­ers with re­li­able cars start to leave the mar­ket, which fur­ther shifts up­ward the prob­a­bil­ity that any given car for sale is a lemon, which makes me less will­ing to pay for a used car, which in­cen­tivizes more hon­est sel­l­ers to leave the mar­ket, and so on.

vis­i­tor: What does the lemons prob­lem have to do with your world’s in­abil­ity to pass around in­for­ma­tion about dead ba­bies?

ce­cie: In our world, there are a lot of peo­ple scream­ing, “Pay at­ten­tion to this thing I’m in­dig­nant about over here!” In fact, there are enough peo­ple scream­ing that there’s an in­ex­ploitable mar­ket in in­dig­na­tion. The dead-ba­bies prob­lem can’t com­pete in that mar­ket; there’s no free en­ergy left for it to eat, and it doesn’t have an op­ti­mal in­dig­na­tion pro­file. There’s no sin­gle in­di­vi­d­ual villain. The busi­ness about com­pet­ing omega-3 and omega-6 metabolic path­ways is some­thing that only a frac­tion of peo­ple would un­der­stand on a visceral level; and even if those peo­ple posted it to their Face­book walls, most of their read­ers wouldn’t un­der­stand and re­post, so the dead-ba­bies prob­lem has rel­a­tively lit­tle viral­ity. Be­ing in­dig­nant about this par­tic­u­lar thing doesn’t sig­nal your moral su­pe­ri­or­ity to any­one else in par­tic­u­lar, so it’s not viscer­ally en­joy­able to en­gage in the in­dig­na­tion. As for adding a fur­ther scream, “But wait, this mat­ter re­ally is im­por­tant!”, that’s the part sub­ject to the lemons prob­lem. Even peo­ple who hon­estly know about a fix­able case of dead ba­bies can’t emit a trust­wor­thy re­quest for at­ten­tion.

sim­pli­cio: You’re say­ing that peo­ple won’t listen even if I sound re­ally in­dig­nant about this? That’s an out­rage!

ce­cie: By this point in our civ­i­liza­tion’s de­vel­op­ment, many hon­est buy­ers and sel­l­ers have left the in­dig­na­tion mar­ket en­tirely; and what’s left be­hind is not, on av­er­age, good.

vis­i­tor: Your re­ply con­tains so many sur­pris­ing pos­tu­lates of weird civ­i­liza­tional dys­func­tion, I hardly know what to ask about next. So in­stead I’ll try to ex­plain how my world works, and you can ex­plain to me why your world doesn’t work that way.

ce­cie: Sounds rea­son­able.

iii. Aca­demic in­cen­tives and beneficiaries

vis­i­tor: To start with, in my world, we have these peo­ple called “sci­en­tists” who ver­ify claims ex­per­i­men­tally, and other peo­ple trust the “sci­en­tists.” So if our “sci­en­tists” say that a cer­tain for­mula seems to be kil­ling ba­bies, this would pro­voke gen­eral in­dig­na­tion with­out ev­ery sin­gle listener need­ing to study do­co­hexa-what­ever acids.

sim­pli­cio: Alas, our so-called sci­en­tists are just pawns of the same med­i­cal-in­dus­trial com­plex that prof­its from kil­ling ba­bies.

ce­cie: I’m afraid, Visi­tor, that al­though there are strong prior rea­sons to ex­pect too much omega-6 and no omega-3 to be very bad for an in­fant baby, and there are now a few dozen small-scale stud­ies which seem to match that pre­dic­tion, this mat­ter hasn’t had the mas­sive study that would be­gin to pro­duce con­fi­dent sci­en­tific agree­ment—

vis­i­tor: You’d bet­ter not be point­ing to that as an ex­oge­nous fact that ex­plains your civ­i­liza­tion’s prob­lem! See, on my planet, if some­body points to strong prior sus­pi­cion com­bined with con­firm­ing pi­lot stud­ies say­ing that some­thing is kil­ling in­no­cent ba­bies and is fix­able, and the pi­lot stud­ies are not con­sid­ered suffi­cient ev­i­dence to set­tle the is­sue, our peo­ple would do more stud­ies and wouldn’t just go on blindly feed­ing the ba­bies poi­son in the mean­time. Our sci­en­tists would all agree on that!

ce­cie: But peo­ple loudly agree­ing on some­thing, by it­self, ac­com­plishes noth­ing. It’s all well and good for ev­ery­one to agree in prin­ci­ple that larger stud­ies ought to be done; but in your world, who ac­tu­ally does the big study, and why do they do it?

vis­i­tor: Two sub­classes within the pro­fes­sion of “sci­en­tist” are sug­gesters, whose pi­lot­ing stud­ies provide the ini­tial sus­pi­cions of effects, and repli­ca­tors whose job it is to con­firm the re­sult and nail things down solidly—the ex­act effect size and so on. When an im­por­tant sug­ges­tive re­sult arises, two repli­ca­tors step for­ward to con­firm it and nail down the ex­act con­di­tions for pro­duc­ing it, be­ing for­bid­den upon their honor to com­mu­ni­cate with each other un­til they sub­mit their find­ings. If both repli­ca­tors agree on the par­tic­u­lars, that com­pletes the dis­cov­ery. The three fund­ing bod­ies that sus­tained the sug­gester and the dual repli­ca­tors would re­ceive the three places of honor in the an­nounce­ment. Do I need to ex­plain how part of the func­tion of any civ­i­lized so­ciety is to ap­pro­pri­ately re­ward those who con­tribute to the pub­lic good?

ce­cie: Well, that’s not how things work on Earth. Our world gives al­most all the pub­lic credit and fame to the dis­cov­erer, as the ini­tial sug­gester is called among us. Our sci­en­tists of­ten say that repli­ca­tion is im­por­tant, but our most pres­ti­gious jour­nals won’t pub­lish mere repli­ca­tions; nor do the his­tory books re­mem­ber them. The out­come is a lot of small stud­ies that have just enough sub­jects to ob­tain “statis­ti­cally sig­nifi­cant” re­sults—

vis­i­tor: … What? Prob­a­bil­ity is quan­ti­ta­tive, not qual­i­ta­tive. There’s no such thing as a “sig­nifi­cant” or “in­signifi­cant” like­li­hood ra­tio—

ce­cie: Any­way, while it might be good if larger stud­ies were done, the de­ci­sion­maker is not the benefi­ciary—the peo­ple who did the ex­tra work of a larger study, and funded the ex­tra work of a larger study, would not re­ceive fame and for­tune thereby.

vis­i­tor: I must be miss­ing some­thing ba­sic here. You do have mul­ti­ple stud­ies, right? When you have mul­ti­ple bod­ies of data, you can mul­ti­ply the like­li­hood func­tions from the stud­ies’ re­spec­tive data to the hy­pothe­ses to ob­tain the mean­ing of the com­bined ev­i­dence—the like­li­hood func­tion from all the data to the hy­pothe­ses.4

ce­cie: I’m afraid you can’t do that on Earth.

vis­i­tor: … Of course you can. It’s a math­e­mat­i­cal the­o­rem. You can’t pos­si­bly tell me that differs be­tween our uni­verses!

Yes, there are pit­falls for the es­pe­cially care­less. Some­times stud­ies end up be­ing con­ducted un­der differ­ent cir­cum­stances, with the re­sult that the naively com­puted like­li­hood func­tions don’t have uniform re­la­tions to the hy­pothe­ses un­der con­sid­er­a­tion. In that case, blindly mul­ti­ply­ing will give you a like­li­hood func­tion that’s nearly zero ev­ery­where. But, I mean, if you just look at all the like­li­hood func­tions, it’s pretty ob­vi­ous when some of them are point­ing in differ­ent di­rec­tions and then you can in­ves­ti­gate that di­ver­gence.

Either it makes sense to mul­ti­ply all the like­li­hood func­tions and get out one mas­sive ev­i­den­tial poin­ter, or else you don’t get a sen­si­ble re­sult when you mul­ti­ply them and then you know some­thing’s wrong with your meth­ods—

ce­cie: I’m afraid our sci­en­tific com­mu­nity doesn’t run on your world’s statis­ti­cal meth­ods. You see, dur­ing the first half of the twen­tieth cen­tury, it be­came con­ven­tional to mea­sure some­thing called “p-val­ues” which im­posed a qual­i­ta­tive dis­tinc­tion be­tween “suc­cess­ful” and “un­suc­cess­ful” ex­per­i­ments—

vis­i­tor: That is still not an ex­pla­na­tion. Why not change the way you do things?

ce­cie: Be­cause some­body who tried us­ing un­con­ven­tional statis­ti­cal meth­ods, even if they were bet­ter statis­ti­cal meth­ods, wouldn’t be able to pub­lish their pa­pers in the most pres­ti­gious jour­nals. And then they wouldn’t get hired. It’s similar to the way that the most pres­ti­gious jour­nals don’t pub­lish mere repli­ca­tions, only dis­cov­er­ies, so peo­ple fo­cus on mak­ing dis­cov­er­ies in­stead of repli­ca­tions.

vis­i­tor: Why would any­one pay at­ten­tion to jour­nals like that?

ce­cie: Be­cause uni­ver­sity hiring de­part­ments care a lot about whether you’ve pub­lished in pres­ti­gious jour­nals.

vis­i­tor: No, I mean… how did these jour­nals end up pres­ti­gious in the first place? Why do uni­ver­sity hiring de­part­ments pay at­ten­tion to them?

sim­pli­cio: Why would uni­ver­sity hiring de­part­ments care about real sci­ence? Shouldn’t it be you who has to ex­plain why some life­less cog of the mil­i­tary-in­dus­trial com­plex would care about any­thing ex­cept grant money?

ce­cie: Okay… you’re dig­ging pretty deep here. I think I need to back up and try to ex­plain things on a more ba­sic level.

vis­i­tor: In­deed, I think you should. So far, ev­ery time I’ve asked you why some­one is act­ing in­sane, you’ve claimed that it’s se­cretly a sane re­sponse to some­one else act­ing in­sane. Where does this pro­cess bot­tom out?

iv. Two-fac­tor mar­kets and sig­nal­ing equilibria

ce­cie: Let me try to iden­tify a first step on which in­san­ity can emerge from non-in­san­ity. Univer­si­ties pay at­ten­tion to pres­ti­gious jour­nals be­cause of a sig­nal­ing equil­ibrium, which, in our tax­on­omy, is a kind of bad Nash equil­ibrium that no sin­gle ac­tor can defy unilat­er­ally.

In your terms, it in­volves a sticky, sta­ble equil­ibrium of ev­ery­one act­ing in­sane in a way that’s se­cretly a sane re­sponse to ev­ery­one else act­ing in­sane.

vis­i­tor: Go on.

ce­cie: First, let me ex­plain the idea of what Eliezer has nick­named a “two-fac­tor mar­ket.” Two-fac­tor mar­kets are a con­cep­tu­ally sim­pler case that will help us later un­der­stand sig­nal­ing equil­ibria.

In our world there’s a crude site for clas­sified ads, called Craigslist. Craigslist doesn’t con­tain any way of rat­ing users, the way that eBay lets buy­ers and sel­l­ers rate each other, or that Airbnb lets renters and land­lords rate each other.

Sup­pose you wanted to set up a ver­sion of Craigslist that let peo­ple rate each other. Would you be able to com­pete with Craigslist?

The an­swer is that even if this in­no­va­tion is in fact a good one, com­pet­ing with Craigslist would be far more difficult than it sounds, be­cause Craigslist is sus­tained by a two-fac­tor mar­ket. The sel­l­ers go where there are the most buy­ers; the buy­ers go where they ex­pect to find sel­l­ers. When you launch your new site, no buy­ers will want to go there be­cause there are no sel­l­ers, and no sel­l­ers will want to go there be­cause there are no buy­ers. Craigslist ini­tially broke into this mar­ket by tar­get­ing San Fran­cisco par­tic­u­larly, and spend­ing mar­ket­ing effort to as­sem­ble the San Fran­cisco buy­ers and sel­l­ers into the same place. But that would be harder to do for a later startup, be­cause now the peo­ple it’s tar­get­ing are already us­ing Craigslist.

sim­pli­cio: Those sheep! Just mind­lessly do­ing what­ever their in­cen­tives tell them to!

ce­cie: We can imag­ine that there’s a bet­ter tech­nol­ogy than Craigslist, called Danslist, such that ev­ery­one us­ing Craigslist would be bet­ter off if they all switched to Danslist si­mul­ta­neously. But if just one buyer or just one sel­ler is the first to go to Danslist, they find an empty park­ing lot. In con­ven­tional cyn­i­cal eco­nomics, we’d say that this is a co­or­di­na­tion prob­lem

sim­pli­cio: A co­or­di­na­tion prob­lem? What do you mean by that?

ce­cie: Back­ing up a bit: A “Nash equil­ibrium” is what hap­pens when ev­ery­one makes their best move, given that all the other play­ers are mak­ing their best moves from that Nash equil­ibrium—ev­ery­one goes to Craigslist, be­cause that’s their in­di­vi­d­u­ally best move given that ev­ery­one else is go­ing to Craigslist. A “Pareto op­ti­mum” is any situ­a­tion where it’s im­pos­si­ble to make ev­ery ac­tor bet­ter off si­mul­ta­neously, like “Co­op­er­ate/​Co­op­er­ate” in the Pri­soner’s Dilemma—there’s no al­ter­na­tive out­come to Co­op­er­ate/​Co­op­er­ate that makes both agents bet­ter off. The Pri­soner’s Dilemma is a co­or­di­na­tion prob­lem be­cause the sole Nash equil­ibrium of Defect/​Defect isn’t Pareto-op­ti­mal; there’s an out­come, Co­op­er­ate/​Co­op­er­ate, that both play­ers pre­fer, but aren’t reach­ing.

sim­pli­cio: How stupid of them!

ce­cie: No, it’s… ah, never mind. Any­way, the frus­trat­ing parts of civ­i­liza­tion are the times when you’re stuck in a Nash equil­ibrium that’s Pareto-in­fe­rior to other Nash equil­ibria. I mean, it’s not sur­pris­ing that hu­mans have trou­ble get­ting to non-Nash op­tima like “both sides co­op­er­ate in the Pri­soner’s Dilemma with­out any other means of en­force­ment or ver­ifi­ca­tion.” What makes an equil­ibrium in­ad­e­quate, a fruit that seems to hang tan­ta­l­iz­ingly low and yet some­how our civ­i­liza­tion isn’t pluck­ing, is when there’s a bet­ter sta­ble state and we haven’t reached it.

vis­i­tor: In­deed. Mov­ing from bad equil­ibria to bet­ter equil­ibria is the whole point of hav­ing a civ­i­liza­tion in the first place.

ce­cie: Be­ing stuck in an in­fe­rior Nash equil­ibrium is how I’d de­scribe the frus­trat­ing as­pect of the two-fac­tor mar­ket of buy­ers and sel­l­ers that can’t switch from Craigslist to Danslist. The sce­nario where ev­ery­one is us­ing Danslist would be a sta­ble Nash equil­ibrium, and a bet­ter Nash equil­ibrium. We just can’t get there from here. There’s no one ac­tor who is be­hav­ing fool­ishly; all the in­di­vi­d­u­als are re­spond­ing strate­gi­cally to their in­cen­tives. It’s only the larger sys­tem that be­haves “fool­ishly.” I’m not aware of a stan­dard term for this situ­a­tion, so I’ll call it an “in­fe­rior equil­ibrium.”

sim­pli­cio: Why do you care what aca­demics call it? Why not just use the best phrase?

ce­cie: The ter­minol­ogy “in­fe­rior equil­ibrium” would be fine if ev­ery­one else were already us­ing that ter­minol­ogy. Mostly I want to use the same phrase that ev­ery­one else uses, even if it’s not the best phrase.

sim­pli­cio: Re­gard­less, I’m not see­ing what the grand ob­sta­cle is to peo­ple solv­ing these prob­lems by, you know, co­or­di­nat­ing. If peo­ple would just act in unity, so much could be done!

I feel like you’re plac­ing too much blame on sys­tem-level is­sues, Ce­cie, when the sim­pler hy­poth­e­sis is just that the peo­ple in the sys­tem are ter­rible: bad at think­ing, bad at car­ing, bad at co­or­di­nat­ing. You claim to be a “cynic,” but your whole world-view sounds rose-tinted to me.

vis­i­tor: Even in my world, Sim­pli­cio, co­or­di­na­tion isn’t as sim­ple as ev­ery­one jump­ing si­mul­ta­neously ev­ery time one per­son shouts “Jump!” For co­or­di­nated ac­tion to be suc­cess­ful, you need to trust the in­sti­tu­tion that says what the ac­tion should be, and a ma­jor­ity of peo­ple have to trust that in­sti­tu­tion, and they have to know that other peo­ple trust the in­sti­tu­tion, so that ev­ery­one ex­pects the co­or­di­nated ac­tion to oc­cur at the crit­i­cal time, so that it makes sense for them to act too.

That’s why we have policy pre­dic­tion mar­kets and… there doesn’t seem to be a word in your lan­guage for the timed-col­lec­tive-ac­tion-thresh­old-con­di­tional-com­mit­ment… hold on, this cul­tural trans­la­tor isn’t mak­ing any sense. “Kick­starter”? You have the key con­cept, but you use it mainly for mak­ing video games?

ce­cie: I’ll now in­tro­duce the con­cept of a sig­nal­ing equil­ibrium.

To para­phrase a com­menter on Slate Star Codex: sup­pose that there’s a mag­i­cal tower that only peo­ple with IQs of at least 100 and some amount of con­scien­tious­ness can en­ter, and this mag­i­cal tower slices four years off your lifes­pan. The nat­u­ral next thing that hap­pens is that em­ploy­ers start to pre­fer prospec­tive em­ploy­ees who have proved they can en­ter the tower, and em­ploy­ers offer these em­ploy­ees higher salaries, or even make en­ter­ing the tower a con­di­tion of be­ing em­ployed at all.5

vis­i­tor: Hold on. There must be less ex­pen­sive ways of test­ing in­tel­li­gence and con­scien­tious­ness than sac­ri­fic­ing four years of your lifes­pan to a mag­i­cal tower.

ce­cie: Let’s not go into that right now. For now, just take as an ex­oge­nous fact that em­ploy­ers can’t get all of the in­for­ma­tion they want by other chan­nels.

vis­i­tor: But—

ce­cie: Any­way: the nat­u­ral next thing that hap­pens is that em­ploy­ers start to de­mand that prospec­tive em­ploy­ees show a cer­tifi­cate say­ing that they’ve been in­side the tower. This makes ev­ery­one want to go to the tower, which en­ables some­body to set up a fence around the tower and charge hun­dreds of thou­sands of dol­lars to let peo­ple in.6

vis­i­tor: But—

ce­cie: Now, for­tu­nately, af­ter Tower One is es­tab­lished and has been run­ning for a while, some­body tries to set up a com­pet­ing mag­i­cal tower, Tower Two, that also drains four years of life but charges less money to en­ter.

vis­i­tor: … You’re solv­ing the wrong prob­lem.

ce­cie: Un­for­tu­nately, there’s a sub­tle way in which this com­pet­ing Tower Two is ham­pered by the same kind of lock-in that pre­vents a jump from Craigslist to Danslist. Ini­tially, all of the smartest peo­ple headed to Tower One. Since Tower One had limited room, it started dis­crim­i­nat­ing fur­ther among its en­trants, only tak­ing the ones that have IQs above the min­i­mum, or who are good at ath­let­ics or have rich par­ents or some­thing. So when Tower Two comes along, the em­ploy­ers still pre­fer em­ploy­ees from Tower One, which has a more fa­mous rep­u­ta­tion. So the smartest peo­ple still pre­fer to ap­ply to Tower One, even though it costs more money. This sta­bi­lizes Tower One’s rep­u­ta­tion as be­ing the place where the smartest peo­ple go.

In other words, the sig­nal­ing equil­ibrium is a two-fac­tor mar­ket in which the sta­ble point, Tower One, is ce­mented in place by the in­di­vi­d­u­ally best choices of two differ­ent parts of the sys­tem. Em­ploy­ers pre­fer Tower One be­cause it’s where the smartest peo­ple go. Smart em­ploy­ees pre­fer Tower One be­cause em­ploy­ers will pay them more for go­ing there. If you try dis­sent­ing from the sys­tem unilat­er­ally, with­out ev­ery­one switch­ing at the same time, then as an em­ployer you end up hiring the less-qual­ified peo­ple from Tower Two, or as an em­ployee, you end up with lower salary offers af­ter you go to Tower Two. So the sys­tem is sta­ble as a mat­ter of in­di­vi­d­ual in­cen­tives, and stays in place. If you try to set up a cheaper al­ter­na­tive to the whole Tower sys­tem, the de­fault thing that hap­pens to you is that peo­ple who couldn’t han­dle the Tow­ers try to go through your new sys­tem, and it ac­quires a rep­u­ta­tion for non-pres­ti­gious weird­ness and in­com­pe­tence.

vis­i­tor: This all just seems so weird and com­pli­cated. I’m skep­ti­cal that this sce­nario with the mag­i­cal tow­ers could hap­pen in real life.

sim­pli­cio: I agree that try­ing to build a cheaper Tower Two is solv­ing the wrong prob­lem. The in­te­rior of Tower One boasts some truly exquisite ar­chi­tec­ture and decor. It just makes sense that some­one should pay a lot to al­low peo­ple en­try to Tower One. What we re­ally need is for the gov­ern­ment to sub­si­dize the en­try fees on Tower One, so that more peo­ple can fit in­side.

ce­cie: Con­sider a sim­pler ex­am­ple: Velcro is a sys­tem for fas­ten­ing shoes that is, for at least some peo­ple and cir­cum­stances, bet­ter than shoelaces. It’s eas­ier to ad­just three sep­a­rate Velcro straps then it is to keep your shoelaces perfectly ad­justed at all loops, it’s faster to do and undo, et cetera, and not ev­ery­one is run­ning at high speeds that call for perfectly ad­justed run­ning shoes. But when Velcro was in­tro­duced, the ear­liest peo­ple to adopt Velcro were those who had the most trou­ble ty­ing their shoelaces—very young chil­dren and the el­derly. So Velcro be­came as­so­ci­ated with kids and old peo­ple, and thus un­for­giv­ably un­fash­ion­able, re­gard­less of whether it would have been bet­ter than shoelaces in some adult ap­pli­ca­tions as well.

vis­i­tor: I take it you didn’t have the stern and up­right lead­ers, what we call the Se­ri­ous Peo­ple, who could set an ex­am­ple by don­ning Velcro shoes them­selves?

sim­pli­cio & ce­cie: (in uni­son) No.

vis­i­tor: I see.

ce­cie: Now con­sider the sys­tem of sci­en­tific jour­nals that we were origi­nally talk­ing about. Some jour­nals are pres­ti­gious. So uni­ver­sity hiring com­mit­tees pay the most at­ten­tion to pub­li­ca­tions in that jour­nal. So peo­ple with the best, most in­ter­est­ing-look­ing pub­li­ca­tions try to send them to that jour­nal. So if a uni­ver­sity hiring com­mit­tee paid an equal amount of at­ten­tion to pub­li­ca­tions in lower-pres­tige jour­nals, they’d end up grant­ing tenure to less pres­ti­gious peo­ple. Thus, the whole sys­tem is a sta­ble equil­ibrium that no­body can unilat­er­ally defy ex­cept at cost to them­selves.

vis­i­tor: I’m still skep­ti­cal. Doesn’t your parable of the mag­i­cal tower sug­gest that, if that’s ac­tu­ally true, some­body ought to rope off the jour­nals too and charge in­sane amounts of money?

ce­cie: Yes, and that’s ex­actly what hap­pened. El­se­vier and a few other prof­i­teers grabbed the most pres­ti­gious jour­nals and started jack­ing up the ac­cess costs. They con­tributed al­most noth­ing—even the peer re­view and edit­ing was done by un­paid vol­un­teers. El­se­vier just charged more and more money and sat back. This is stan­dardly called rent-seek­ing. In a few cases, the sci­en­tists were able to kick­start a co­or­di­nated move where the en­tire edit­ing board would re­sign, start a new jour­nal, and ev­ery­body in the field would sub­mit to the new jour­nal in­stead. But since our sci­en­tists don’t have rec­og­nized kick­start­ing cus­toms, or any soft­ware sup­port for them, it isn’t easy to pull that off. Most of the big-name jour­nals that El­se­vier has cap­tured are still big names, still get­ting pres­ti­gious sub­mis­sions, and still cap­tur­ing big-money rents.

vis­i­tor: Well, I guess I un­der­stand why my cul­tural trans­la­tor keeps putting air quotes around Earth’s ver­sion of “sci­ence.” The whole idea of sci­ence, as I un­der­stand the con­cept, is that ev­ery­thing has to be in the open for any­one to ver­ify. Science is the part of hu­man­ity’s knowl­edge that ev­ery­one can po­ten­tially learn about and re­pro­duce them­selves. You can’t charge money in or­der for peo­ple to read your ex­per­i­men­tal re­sults, or you lose the “ev­ery­one can ac­cess and ver­ify your claims” prop­erty that dis­t­in­guishes sci­ence from other kinds of in­for­ma­tion.

ce­cie: Oh, rest as­sured that sci­en­tists aren’t see­ing any of this money. It all goes to the third-party jour­nal own­ers.

sim­pli­cio: And this isn’t just sci­en­tists be­ing stupid?

ce­cie: No stupi­der than you are for go­ing to col­lege. It’s hard to beat sig­nal­ing equil­ibria—be­cause they’re “multi-fac­tor mar­kets”—which are spe­cial cases of co­or­di­na­tion prob­lems that cre­ate “in­fe­rior Nash equil­ibria”—which are so stuck in place that mar­ket con­trol­lers can seek rent on the value gen­er­ated by cap­tive par­ti­ci­pants.

sim­pli­cio: Weren’t we talk­ing about dead ba­bies at some point?

ce­cie: Yes, we were. I was ex­plain­ing how our sys­tem al­lo­cated too much credit to dis­cov­er­ers and not enough credit to repli­ca­tors, and the only so­cially ac­cept­able statis­tics couldn’t ag­gre­gate small-scale tri­als in a way re­garded as re­li­able. The Visi­tor asked me why the sys­tem was like that. I pointed to jour­nals that pub­lished a par­tic­u­lar kind of pa­per. The Visi­tor asked me why any­one paid at­ten­tion to those jour­nals in the first place. I ex­plained about sig­nal­ing equil­ibria, and that’s where we are now.

vis­i­tor: I can’t say that I feel en­light­ened at the end of walk­ing through all that. There must be par­tic­u­lar sci­en­tists on the ed­i­to­rial boards who choose not to de­mand repli­ca­tions and who for­bid mul­ti­ply­ing like­li­hood ra­tios. Why are those par­tic­u­lar sci­en­tists do­ing the non-sen­si­ble thing?

ce­cie: Be­cause peo­ple in the gen­eral field wouldn’t cite non­stan­dard pa­pers, so if the ed­i­tors de­manded non­stan­dard pa­pers, the jour­nal’s pub­li­ca­tion fac­tor would de­crease.

vis­i­tor: Why don’t the jour­nal ed­i­tors start by de­mand­ing that pa­per sub­mit­ters cite dual repli­ca­tions as well as ini­tial sug­ges­tions?

ce­cie: Be­cause that would be a weird un­con­ven­tional de­mand, which might lead peo­ple with high-pres­tige re­sults to sub­mit those re­sults to other jour­nals in­stead. Fun­da­men­tally, you’re ask­ing why sci­en­tists on Earth don’t adopt cer­tain new cus­toms that you think would be for the good of ev­ery­one. And the an­swer is that there’s this big, multi-fac­tor sys­tem that no­body can dis­sent from unilat­er­ally, and that peo­ple have a lot of trou­ble co­or­di­nat­ing to change. That’s true even when there are forces like El­se­vier that are be­ing blatant about rip­ping ev­ery­one off. Im­ple­ment­ing your pro­posed cul­tural shift to “sug­gesters” and “repli­ca­tors,” or us­ing like­li­hood func­tions, would be sig­nifi­cantly harder than ev­ery­one just si­mul­ta­neously ceas­ing to deal with El­se­vier, since the case for it would be less ob­vi­ous and would pro­voke more dis­agree­ment. All that we can man­age is to make in­cre­men­tal shifts to­ward fund­ing more repli­ca­tion and ask­ing more for study pre­reg­is­tra­tion.

To sum up, aca­demic sci­ence is em­bed­ded in a big enough sys­tem with enough sep­a­rate de­ci­sion­mak­ers cre­at­ing in­cen­tives for other de­ci­sion­mak­ers that it al­most always takes the path of least re­sis­tance. The sys­tem isn’t in the best Nash equil­ibrium be­cause no­body has the power to look over the sys­tem and choose good Nash equil­ibria. It’s just in a Nash equil­ibrium that it wan­dered into, which in­cludes statis­ti­cal meth­ods that were in­vented in the first half of the 20th cen­tury and ed­i­tors not de­mand­ing that peo­ple cite repli­ca­tions.

vis­i­tor: I see. And that’s why no­body in your world has mul­ti­plied the like­li­hood func­tions, or done a large-enough sin­gle study, or oth­er­wise done what­ever it would take to con­vince who­ever needs to be con­vinced about the effects of feed­ing in­fants soy­bean oil.

ce­cie: It’s one of the rea­sons. A large study would also be very ex­pen­sive be­cause of ex­treme pa­per­work re­quire­ments, gen­er­ated by other sys­temic failures I haven’t got­ten around to talk­ing about yet—7

vis­i­tor: How does any­thing get done ever, in your world?

ce­cie: —and when it comes to fund­ing or car­ry­ing out that big­ger study, the de­ci­sion­maker would not sig­nifi­cantly benefit un­der the cur­rent sys­tem, which is held in place by co­or­di­na­tion prob­lems. And that’s why peo­ple who already have a back­ground grasp of lipid metabolic path­ways have asym­met­ric in­for­ma­tion about what is worth be­com­ing in­dig­nant about.

v. To­tal mar­ket failures

vis­i­tor: Even grant­ing the things you’ve said already, I don’t feel like I’ve been told enough to un­der­stand why your so­ciety is kil­ling ba­bies.

ce­cie: Well, no. Not yet. The lack of in­cen­tive to do a large-scale con­vinc­ing study is only one thing that went wrong in­side one part of the sys­tem. There’s a lot more bro­ken than just that—which is why effec­tive al­tru­ists shouldn’t be run­ning out and try­ing to fund a big repli­ca­tion study for Omegaven, be­cause that by it­self wouldn’t fix things.

vis­i­tor: Okay, sup­pose there had been a large enough study to satisfy your world’s take on “sci­en­tists.” What else would likely go wrong af­ter that?

ce­cie: Sev­eral things. For ex­am­ple, doc­tors wouldn’t nec­es­sar­ily be aware of the ex­per­i­men­tal re­sults.

vis­i­tor: Hold on, I think my cul­tural trans­la­tor is bro­ken. You used that word “doc­tor” and my trans­la­tor spit out a long se­quence of words for Ex­am­iner plus Di­ag­nos­ti­cian plus Treat­ment Plan­ner plus Sur­geon plus Out­come Eval­u­a­tor plus Stu­dent Trainer plus Busi­ness Man­ager. Maybe it’s stuck and spit­ting out the names of all the pro­fes­sions as­so­ci­ated with medicine.

ce­cie: So, in your world, if there is a dual repli­ca­tion of re­sults on Omegaven ver­sus soy­bean oil, how does that end up chang­ing the ac­tual pa­tient treat­ments?

vis­i­tor: By in­form­ing the Treat­ment Plan­ners who spe­cial­ize in in­fant ail­ments that re­quired par­enteral nu­tri­tion, of course. The dis­cov­ery would ap­pear in­side the “par­enteral nu­tri­tion” pages in the Earth­web and show up in the feeds of ev­ery­one sub­scribed to that page. The statis­tics would ap­pear in­side the Treat­ment Plan­ner’s de­ci­sion-sup­port soft­ware. And if all of those broke for some rea­son, ev­ery Treat­ment Plan­ner for in­fant ail­ments that re­quired par­enteral nu­tri­tion would just use cha­t­rooms. And any­one who ig­nored the cha­t­rooms would have worse pa­tient out­come rat­ings, and would lose sta­tus rel­a­tive to Treat­ment Plan­ners who were more at­ten­tive.

ce­cie: It sounds like “Treat­ment Plan­ners” in your world are much more spe­cial­ized than doc­tors in this world. I sup­pose they’re also se­lected speci­fi­cally for tal­ent at… cost-benefit anal­y­sis and de­ci­sion the­ory, or some­thing along those lines? And then they fo­cus their learn­ing on par­tic­u­lar dis­eases for which they are Treat­ment Plan­ners? And some­body else tracks their out­comes?

vis­i­tor: Of course. I’m… al­most afraid to ask, but how do they do it in your world?

ce­cie: Your trans­la­tor wasn’t bro­ken. In our world, “doc­tors” are sup­posed to ex­am­ine pa­tients for symp­toms, di­ag­nose es­pe­cially com­pli­cated or ob­scure ail­ments us­ing their en­cy­clo­pe­dic knowl­edge and their keen grasp of Bayesian in­fer­ence, plan the pa­tient’s treat­ment by weigh­ing the costs and benefits of the lat­est treat­ments, ex­e­cute the treat­ments us­ing their keen dex­ter­ity and re­li­able stamina, eval­u­ate for them­selves how well that went, train stu­dents to do it too, and in many cases, also over­see the small busi­ness that bills the pa­tients and mar­kets it­self. So “doc­tors” have to be se­lected for all of those tal­ents si­mul­ta­neously, and then split their train­ing, ex­pe­rience, and at­ten­tion be­tween them.

vis­i­tor: Why in the name of—

ce­cie: Oh, and be­fore they go to med­i­cal school, we usu­ally send them off to get a four-year de­gree in philos­o­phy first or some­thing, just be­cause.

I don’t know if there’s a stan­dard name for this phe­nomenon, but we can call it “failure of pro­fes­sional spe­cial­iza­tion.” It also ap­pears when, for ex­am­ple, a lawyer has to learn calcu­lus in or­der to grad­u­ate col­lege, even though their job doesn’t re­quire any calcu­lus.

vis­i­tor: Why. Why. Why why why—

ce­cie: I’m not sure. I sus­pect the ori­gin has some­thing to do with sta­tus—like, a high-sta­tus per­son can do all things at once, so it’s in­sult­ing and low­ers sta­tus to sug­gest that an es­teemed and re­spectable Doc­tor should only prac­tice one sur­gi­cal op­er­a­tion and get very good at it. And once you your­self have spent twelve years be­ing trained un­der the cur­rent sys­tem, you won’t be happy about the pro­posal to re­place it with two years of much more spe­cial­ized train­ing. Once you’ve been through a painful ini­ti­a­tion rit­ual and ra­tio­nal­ized its ne­ces­sity, you’ll hate to see any­one else go­ing through a less painful one. Not to men­tion that you won’t be happy about the com­pe­ti­tion against your own hu­man cap­i­tal, by a cheaper and bet­ter form of hu­man cap­i­tal—and af­ter the sunk cost in pain and time that you en­dured to build hu­man cap­i­tal un­der the old sys­tem…

vis­i­tor: Do they not have mar­kets on your planet? Be­cause on my planet, when you man­u­fac­ture your product in a crazy, elab­o­rate, ex­pen­sive way that pro­duces an in­fe­rior product, some­one else will come along and ra­tio­nal­ize the pro­cess and take away your cus­tomers.

ce­cie: We have mar­kets, but there’s this un­for­tu­nate thing called “reg­u­la­tory cap­ture,” of which one kind is “oc­cu­pa­tional li­cens­ing.”

As an ex­am­ple, it used to be that chairs were care­fully hand-crafted one at the time by car­pen­ters who had to un­dergo a lengthy ap­pren­tice­ship, and in­deed, they didn’t like it when fac­to­ries came along staffed by peo­ple who spe­cial­ized in just carv­ing a sin­gle kind of arm. But the fac­tory-made chairs were vastly cheaper and most of the peo­ple who in­sisted on stick­ing to hand­crafts soon went out of busi­ness.

Now imag­ine: What if the chair-mak­ers had been ex­tremely re­spectable—had already pos­sessed very high sta­tus? What if their pro­fes­sion had an el­e­ment of dan­ger? What if they’d man­aged to frighten ev­ery­one about the dan­gers of im­prop­erly made chairs that might dump peo­ple on the ground and snap their necks?

vis­i­tor: Okay, yes, we used to have Se­ri­ous Peo­ple who would go around and cer­tify the mak­ing of some medicines where some­body might be tempted to cheat and use in­fe­rior in­gre­di­ents. But that was be­fore com­put­ers and out­come statis­tics and on­line rat­ings.

ce­cie: And on our planet, Uber and Lyft are cur­rently fight­ing it out with taxi com­pa­nies and their pet reg­u­la­tors af­ter ex­actly that de­vel­op­ment. But sup­pose the whole sys­tem was set up be­fore the ex­is­tence of on­line rat­ings. Then the car­pen­ters might have man­aged to in­tro­duce oc­cu­pa­tional li­cens­ing on who could be a car­pen­ter. So if you tried to set up a fac­tory, your fac­tory work­ers would have needed to go through the tra­di­tional car­pen­try ap­pren­tice­ship that cov­ered ev­ery part of ev­ery kind of fur­ni­ture, be­fore they were legally al­lowed to come to your fac­tory and spe­cial­ize in carv­ing just one kind of chair-arm. And then your fac­tory would also need a ton of per­mits to sell its fur­ni­ture, and would need to in­vei­gle or­ders from a hand­ful of re­sel­lers who were li­censed to buy and re­sell fur­ni­ture at a fixed mar­gin. That small, in­su­lar group of re­sel­lers might not benefit liter­ally per­son­ally—in their own per­sonal salary—from buy­ing from your cheaper fac­tory sys­tem. And so it would go.

vis­i­tor: But why would the leg­is­la­tors go along with that?

ce­cie: Be­cause the car­pen­ters would have a big, con­cen­trated in­cen­tive to figure out how to make leg­is­la­tors do it—maybe by hiring very per­sua­sive peo­ple, or by sub­tle bribery, or by not-so-sub­tle bribery.

In­so­far as oc­cu­pa­tional li­cens­ing works to the benefit of pro­fes­sion­als at the ex­pense of con­sumers, oc­cu­pa­tional li­cens­ing rep­re­sents a kind of reg­u­la­tory cap­ture, which hap­pens when a few reg­u­la­tees have a much more con­cen­trated in­cen­tive to af­fect the reg­u­la­tion pro­cess. Reg­u­la­tory cap­ture in turn is a kind of com­mons prob­lem, since ev­ery cit­i­zen shares the benefits of non-cap­tured reg­u­la­tion, but no in­di­vi­d­ual cit­i­zen has a suffi­cient in­cen­tive to unilat­er­ally spend their life at­tend­ing to that par­tic­u­lar reg­u­la­tory prob­lem. So oc­cu­pa­tional li­cens­ing is reg­u­la­tory cap­ture is a com­mons prob­lem is a co­or­di­na­tion prob­lem.

vis­i­tor: Then… the up­shot is that it’s im­pos­si­ble for your coun­try to test a func­tional hos­pi­tal de­sign in the first place? The re­form­ers can’t win the com­pe­ti­tion be­cause they’re not legally al­lowed to try?

ce­cie: But of course. Though in this case, if you did man­age to set up a test hos­pi­tal work­ing along more rea­son­able lines, you still wouldn’t be able to ad­ver­tise your bet­ter re­sults rel­a­tive to any other hos­pi­tals. With just a few iso­lated ex­cep­tions, all of the other hos­pi­tals on Earth don’t pub­lish pa­tient out­come statis­tics in the first place.

vis­i­tor: … But… then—what are they even sel­l­ing?

sim­pli­cio: Hold on. If you re­ward the doc­tors with the high­est pa­tient sur­vival rates, won’t they just re­ject all the pa­tients with poor prog­noses?

vis­i­tor: Ob­vi­ously you don’t eval­u­ate raw sur­vival rates. You have Di­ag­nos­ti­ci­ans who es­ti­mate prog­no­sis cat­e­gories and are rated on their pre­dic­tive ac­cu­racy, and Treat­ment Plan­ners and Sur­geons who are rated on their rel­a­tive out­comes, and you have the out­comes eval­u­ated by a third party, and—

ce­cie: In our world, there’s no sep­a­ra­tion of pow­ers where one per­son as­signs pa­tients a prog­no­sis cat­e­gory and has their pre­dic­tion record tracked, and an­other per­son does their best to treat them and has their treat­ment record tracked. So hos­pi­tals don’t pub­lish any perfor­mance statis­tics, and pa­tients choose the hos­pi­tal clos­est to their house that takes their work­place’s in­surance, and no­body has any fi­nan­cial in­cen­tive to de­crease the num­ber of pa­tient deaths from sloppy sur­geons or cen­tral line in­fec­tions. When anes­the­siol­o­gists in par­tic­u­lar did hap­pen to start track­ing pa­tient out­comes, they adopted some sim­ple mon­i­tor­ing stan­dards and sub­se­quently de­creased their fatal­ity rates by a fac­tor of one hun­dred.8 But that’s just anes­the­siol­o­gists, not, say, car­diac sur­geons.

With car­diac sur­geons, a group of re­searchers re­cently figured out how to de­tect when the most se­nior car­diac sur­geons were at con­fer­ences, and found that the death rates went down while the most se­nior car­diac sur­geons were away.9 But our sci­en­tists have to use spe­cial tricks if they want to find out any facts like that.

vis­i­tor: Do your pa­tients not care if they live or die?

ce­cie: Robin Han­son has a fur­ther the­sis about how what peo­ple re­ally want from medicine is re­as­surance rather than statis­tics. But I’m not sure that hy­poth­e­sis is nec­es­sary to ex­plain this par­tic­u­lar as­pect of the prob­lem. If no hos­pi­tal offers statis­tics, then you have no baseline to com­pare to if one hos­pi­tal does start offer­ing statis­tics. You’d just be look­ing at an alarm­ing-look­ing per­centage for how many pa­tients die, with no idea of whether that’s a bet­ter per­centage or a worse per­centage. Ter­rible mar­ket­ing! Espe­cially com­pared to that other hos­pi­tal across town that just smiles at you re­as­sur­ingly.

No hos­pi­tal would benefit from be­ing the first to pub­lish statis­tics, so none of them do.

vis­i­tor: Your world has liter­ally zero mar­ket de­mand for em­piri­cal ev­i­dence?

ce­cie: Not zero, no. But since pub­lish­ing scary num­bers would be bad mar­ket­ing for most pa­tients, and hos­pi­tals are heav­ily re­gional, they all go by the ma­jor­ity prefer­ence to not hear about the statis­tics.

vis­i­tor: I con­fess I’m hav­ing some trou­ble grasp­ing the con­cept of a mar­ket con­sist­ing of opaque boxes allegedly con­tain­ing goods, in which no­body pub­lishes what is in­side the boxes.

ce­cie: Hospi­tals don’t pub­lish prices ei­ther, in most cases.

vis­i­tor:

ce­cie: Yeah, it’s pretty bad even by Earth stan­dards.

vis­i­tor: You liter­ally don’t have a health­care mar­ket. No­body knows what out­comes are be­ing sold. No­body knows what the prices are.

ce­cie: I guess we could call that To­tal Mar­ket Failure? As in, things have gone so wrong that there’s liter­ally no sup­ply-de­mand match­ing or price-equil­ibrat­ing mechanism re­main­ing, even though money is still chang­ing hands.

And while I wish that this phe­nomenon of “you sim­ply don’t have a mar­ket” were only rele­vant to health­care and not to other facets of our civ­i­liza­tion… well, it’s not.

vi. Ab­sence of (meta-)competition

vis­i­tor: I sup­pose I can imag­ine imag­ine a hy­po­thet­i­cal world in which one coun­try screws things up as badly as you de­scribe. But your planet has mul­ti­ple gov­ern­ments, I thought. Or did I mi­s­un­der­stand that? Why wouldn’t pa­tients em­i­grate to—or just visit—coun­tries that made bet­ter hos­pi­tals le­gal?

ce­cie: The forces act­ing on gov­ern­ments with high tech­nol­ogy lev­els are mostly the same be­tween coun­tries, so all the gov­ern­ments of those coun­tries tend to have their med­i­cal sys­tem screwed up in mostly the same way (not least be­cause they’re imi­tat­ing each other). Some as­pects of dys­func­tional in­surance and pay­ment poli­cies are spe­cial to the US, but even the rel­a­tively func­tional Na­tional Health Sys­tem in Bri­tain still has failure of pro­fes­sional spe­cial­iza­tion. (Though they at least don’t re­quire doc­tors to have philos­o­phy de­grees.)

vis­i­tor: Is there not one gov­ern­ment that would al­low a rea­son­ably de­signed hos­pi­tal staffed by spe­cial­ists in­stead of gen­er­al­ists?

ce­cie: It wouldn’t be enough to just have one gov­ern­ment’s okay. You’d need some way to ini­tially train your work­ers, de­spite none of our world’s med­i­cal schools be­ing set up to train them. A ma­jor­ity of leg­is­la­tors won’t benefit per­son­ally from de­cid­ing to let you try your new hos­pi­tal in their coun­try. Fur­ther­more, you couldn’t just go around rais­ing money from rich coun­tries for a ven­ture in a poor coun­try, be­cause rich coun­tries have elab­o­rate reg­u­la­tions on who’s al­lowed to raise money for busi­ness ven­tures through equity sales. The fun­da­men­tal story is that ev­ery­thing, ev­ery­where, is cov­ered with vary­ing de­grees of mo­lasses, and to do any novel thing you have to get around all of the mo­lasses streams si­mul­ta­neously.

vis­i­tor: So it’s im­pos­si­ble to test a func­tional hos­pi­tal de­sign any­where on the planet?

ce­cie: But of course.

vis­i­tor: I must still be miss­ing some­thing. I just don’t un­der­stand why all of the peo­ple with eco­nomics train­ing on your planet can’t go off by them­selves and es­tab­lish their own hos­pi­tals. Do you liter­ally have peo­ple oc­cu­py­ing ev­ery square mile of land?

ce­cie: … How do I phrase this…

All use­ful land is already claimed by some na­tional gov­ern­ment, in a way that the in­ter­na­tional or­der rec­og­nizes, whether or not that land is in­hab­ited. No rele­vant de­ci­sion­maker has a per­sonal in­cen­tive to al­low there to be un­claimed land. Those coun­tries will defend even a very small patch of that claimed land us­ing all of the mil­i­tary force their coun­try has available, and the in­ter­na­tional or­der will see you as the ag­gres­sor in that case.

vis­i­tor: Can you buy land?

ce­cie: You can’t buy the sovereignty on the land. Even if you had a lot of money, any coun­try poor enough and des­per­ate enough to con­sider your offer might just steal your stuff af­ter you moved in.

Ne­go­ti­at­ing the right to bring in weapons to defend your­self in this kind of sce­nario would be even more un­think­able, and would spark in­ter­na­tional out­rage that could pre­vent you from trad­ing with other coun­tries.

To be clear, it’s not that there’s a global dic­ta­tor who pre­vents new coun­tries from pop­ping up; but ev­ery po­ten­tially use­ful part of ev­ery land is un­der some sys­tem’s con­trol, and all of those sys­tems would re­fuse you the chance to set up your own al­ter­na­tive sys­tem, for very similar rea­sons.

vis­i­tor: So there’s no way for your planet to try differ­ent ways of do­ing things, any­where. You liter­ally can­not run ex­per­i­ments about things like this.

ce­cie: Why would there be? Who would de­cide that, and how would they per­son­ally benefit?

vis­i­tor: That sounds ex­tremely alarm­ing. I mean, difficul­ties of adop­tion are one thing, but not even be­ing able to try new things and see what hap­pens… Shouldn’t ev­ery­one on your planet be able to de­tect at a glance how hor­rible things have be­come? Can this type of dis­aster re­ally stand up to uni­ver­sal agree­ment that some­thing is wrong?

ce­cie: I’m afraid that our civ­i­liza­tion doesn’t have a suffi­ciently stir­ring and nar­ra­tively satis­fy­ing con­cep­tion of the valor of “test­ing things” that our peo­ple would be mas­sively alarmed by its im­pos­si­bil­ity. And now, Visi­tor, I hope we’ve bot­tomed out the gen­eral con­cept of why peo­ple can’t do things differ­ently—the lo­cal sys­tem’s equil­ibrium is bro­ken, and the larger sys­tem’s equil­ibrium makes it im­pos­si­ble to flee the game.

vis­i­tor: Okay, look… de­spite ev­ery­thing you’ve said so far, I still have some trou­ble un­der­stand­ing why doc­tors and par­ents can’t just not kill the ba­bies. I man­age to get up ev­ery sin­gle morn­ing and suc­cess­fully not kill any ba­bies. It’s not as hard as it sounds.

ce­cie: I worry you’re start­ing to think like Sim­pli­cio. You can’t just not kill ba­bies and ex­pect to get away with it.

sim­pli­cio: I ac­tu­ally agree with Ce­cie here. The evil peo­ple be­hind the sys­tem hate those who defy them by be­hav­ing differ­ently; there’s no way they’d coun­te­nance any­one de­part­ing from the norm. What we re­ally need is a rev­olu­tion, so we can de­pose our cor­rupt over­lords, and fi­nally be free to co­or­di­nate, and…!

ce­cie: There’s no need to add in any evil con­spir­acy hy­pothe­ses here.

It’s suffi­cient to note that the sys­tem is in equil­ibrium and it has causes for the equil­ibrium set­tling there—causes, if not jus­tifi­ca­tions. You can’t go against the sys­tem’s de­fault with­out go­ing against the forces that un­der­pin that de­fault. A doc­tor who gives a baby a nu­tri­tion for­mula that isn’t FDA-ap­proved will lose their job. A hos­pi­tal that doesn’t fire that kind of doc­tor will be sued. A sci­en­tist that writes pro­pos­als for a big, ex­pen­sive, defini­tive study won’t get a grant, and while they were busy writ­ing those failed grant pro­pos­als, they’ll have lost their mo­men­tum to­ward tenure. So no, you can’t just try out a com­pet­ing policy of not kil­ling ba­bies. Not more than once.

vis­i­tor: Have you tried?

ce­cie: No.

vis­i­tor: But—

ce­cie: Any­way, from my per­spec­tive, it’s no sur­prise if you don’t yet feel like you un­der­stand. We’ve only be­gun to sur­vey the malfunc­tions of the whole sys­tem, which would fur­ther in­clude the FDA, and the clini­cal tri­als, and the p-hack­ing. And the way ven­ture cap­i­tal is struc­tured, and equity-mar­ket reg­u­la­tions. And the in­surance com­pa­nies, and the tax code. And the cor­po­ra­tions who con­tract with the in­surance com­pa­nies. And the cor­po­ra­tions’ em­ploy­ees. And the poli­ti­ci­ans. And the vot­ers.

vis­i­tor: … Con­sider me im­pressed that your planet man­aged to reach this level of dys­func­tion with­out ac­tu­ally phys­i­cally burst­ing into flames.


Next: Moloch’s Toolbox part 2.

The full book will be available Novem­ber 16th. You can go to equil­ibri­abook.com to pre-or­der the book, or sign up for no­tifi­ca­tions about new chap­ters and other de­vel­op­ments.


  1. Carl Shul­man notes that the Afford­able Care Act linked fed­eral pay­ments to hos­pi­tals with re­duc­ing cen­tral-line in­fec­tions (source), which was prob­a­bly a fac­tor in the change.

  2. Around a thou­sand in­fants are born with short bowel syn­drome per year in the United States, of whom two-thirds de­velop par­enteral nu­tri­tion-as­so­ci­ated liver dis­ease (source). See Park, Ne­s­por, and Kerner Jr for a 2011 re­view of the aca­demic liter­a­ture, and Koch, Co­hen, and Car­roll and Madrzyk for news cov­er­age.

  3. See Tabar­rok’s “Assess­ing the FDA via the Ano­maly of Off-La­bel Drug Pre­scribing,” which cites the wide­spread prac­tice of off-la­bel pre­scrip­tion as ev­i­dence that the FDA’s effi­cacy trial re­quire­ments are un­nec­es­sary.

  4. See the “Re­port Like­li­hoods, Not p-Values” FAQ, or, in di­alogue form: “Like­li­hood Func­tions, p-Values, and the Repli­ca­tion Cri­sis.”

  5. From Sch­midt and Hunter’s “Select on In­tel­li­gence”: “In­tel­li­gence is the ma­jor de­ter­mi­nant of job perfor­mance, and there­fore hiring peo­ple based on in­tel­li­gence leads to marked im­prove­ments in job perfor­mance.” See also psy­chol­o­gist Stu­art Ritchie’s dis­cus­sion of IQ in Vox.

    Soft­ware en­g­ineer Alyssa Vance adds:

    I’ll note that, as far as I can tell, the in­for­mal con­sen­sus at least among the best-in­formed peo­ple in soft­ware is that hiring has tons of ob­vi­ous ir­ra­tional­ity even when there’s definitely no ex­ter­nal cause; see [1] and [2]. In terms of Moloch’s toolbox, the ob­vi­ous rea­son for that is that in­ter­view­ers are rarely judged on the qual­ity of the peo­ple they ac­cept, and when they are, cer­tainly aren’t paid more or less based on it. (Never mind the peo­ple they re­ject. “No­body ever got fired be­cause of the later perfor­mance of some­one they turned down.”) Their in­cen­tive, in­so­far as they have one, is to hire peo­ple who they’d most pre­fer to be on the same floor with all day long.

  6. Com­pare psy­chi­a­trist Scott Alexan­der’s ac­count, in “Against Tulip Sub­sidies”:

    In Amer­ica, as­piring doc­tors do four years of un­der­grad in what­ever area they want (I did Philos­o­phy), then four more years of med­i­cal school, for a to­tal of eight years post-high school ed­u­ca­tion. In Ire­land, as­piring doc­tors go straight from high school to med­i­cal school and finish af­ter five years. I’ve done medicine in both Amer­ica and Ire­land. The doc­tors in both coun­tries are about equally good. When Ir­ish doc­tors take the Amer­i­can stan­dard­ized tests, they usu­ally do pretty well. Ire­land is one of the ap­prox­i­mately 100% of First World coun­tries that gets bet­ter health out­comes than the United States. There’s no ev­i­dence what­so­ever that Amer­i­can doc­tors gain any­thing from those three ex­tra years of un­der­grad. And why would they? Why is hav­ing a philos­o­phy de­gree un­der my belt sup­posed to make me any bet­ter at medicine? […]

    I’ll make an­other con­fes­sion. Ire­land’s med­i­cal school is five years as op­posed to Amer­ica’s four be­cause the Ir­ish spend their first year teach­ing the ba­sic sci­ences—biol­ogy, or­ganic chem­istry, physics, calcu­lus. When I ap­plied to med­i­cal school in Ire­land, they offered me an ac­cel­er­ated four year pro­gram on the grounds that I had surely got­ten all of those in my Amer­i­can un­der­grad­u­ate work. I hadn’t. I read some books about them over the sum­mer and did just fine.

    Amer­i­cans take eight years to be­come doc­tors. Ir­ish­men can do it in four, and achieve the same re­sult. Each year of higher ed­u­ca­tion at a good school—let’s say an Ivy, doc­tors don’t study at Po­dunk Com­mu­nity Col­lege—costs about $50,000. So Amer­i­can med­i­cal stu­dents are pay­ing an ex­tra $200,000 for…what?

    Re­mem­ber, a mod­est amount of the cur­rent health care crisis is caused by doc­tors’ crip­pling level of debt. So­cially re­spon­si­ble doc­tors of­ten con­sider less lu­cra­tive ca­reers helping the needy, right up un­til the bill comes due from their ed­u­ca­tion and they re­al­ize they have to make a lot of money right now. We took one look at that prob­lem and said “You know, let’s make doc­tors pay an ex­tra $200,000 for no rea­son.”

    For a more gen­eral dis­cus­sion of the ev­i­dence that col­lege is chiefly a costly sig­nal of pre-ex­ist­ing abil­ity, rather than a mechanism for build­ing skills and im­prov­ing pro­duc­tivity, see Bryan Ca­plan’s ar­gu­ment in “Is Col­lege Worth It?”, also sum­ma­rized by Roger Bar­ris.

  7. See, e.g., Scott Alexan­der’s “My IRB Night­mare.”

  8. From Hy­man and Silver, “You Get What You Pay For”:

    By the 1950s, death rates ranged be­tween 1 and 10 per 10,000 en­coun­ters. Anes­the­sia mor­tal­ity sta­bi­lized at this rate for more than two decades. Mor­tal­ity and mor­bidity rates fell again af­ter a 1978 ar­ti­cle re­framed the is­sue of anes­the­sia safety as one of hu­man fac­tor anal­y­sis. In the mid-1980s, the Amer­i­can So­ciety of Anes­the­siol­o­gists (ASA) pro­mul­gated stan­dards of op­ti­mal anes­the­sia prac­tice that re­lied heav­ily on sys­tems-based ap­proaches for pre­vent­ing er­rors. Be­cause pa­tients fre­quently sued anes­thetists when bad out­comes oc­curred and be­cause de­vi­a­tions from the ASA guidelines made the im­po­si­tion of li­a­bil­ity much more likely, anes­thetists had sub­stan­tial in­cen­tives to com­ply.

    [… W]e should con­sider why anes­the­sia mor­tal­ity sta­bi­lized at a rate more than one hun­dred times higher than its cur­rent level for more than two decades. The prob­lem was not lack of in­for­ma­tion. To the con­trary, anes­the­sia safety was stud­ied ex­ten­sively dur­ing the pe­riod. A bet­ter hy­poth­e­sis is that anes­thetists grew ac­cus­tomed to a mor­tal­ity rate that was ex­em­plary by health care stan­dards, but that was still higher than it should have been. From a psy­cholog­i­cal per­spec­tive, this low fre­quency en­couraged anes­thetists to treat each bad out­come as a tragic but un­fore­seen and un­pre­ventable event. In­deed, anes­thetists likely viewed each in­di­vi­d­ual bad out­come as the man­i­fes­ta­tion of an ir­re­ducible baseline rate of med­i­cal mishap.

    Hy­man and Silver note other pos­si­ble fac­tors be­hind the large change, e.g., the fact that the per­son re­spon­si­ble for mishaps was of­ten easy to iden­tify since there tended to be only one anes­thetist per pro­ce­dure, and that “be­cause sur­gi­cal pa­tients had no on-go­ing re­la­tion­ships with their anes­thetist, vic­tims were par­tic­u­larly likely to sue.”

  9. See Jena, Prasad, Gold­man, and Rom­ley, “Mor­tal­ity and Treat­ment Pat­terns Among Pa­tients Hospi­tal­ized With Acute Car­dio­vas­cu­lar Con­di­tions Dur­ing Dates of Na­tional Car­diol­ogy Meet­ings.”