Bayesians vs. Barbarians


Let’s say we have two groups of sol­diers. In group 1, the pri­vates are ig­no­rant of tac­tics and strat­egy; only the sergeants know any­thing about tac­tics and only the officers know any­thing about strat­egy. In group 2, ev­ery­one at all lev­els knows all about tac­tics and strat­egy.

Should we ex­pect group 1 to defeat group 2, be­cause group 1 will fol­low or­ders, while ev­ery­one in group 2 comes up with bet­ter ideas than what­ever or­ders they were given?

In this case I have to ques­tion how much group 2 re­ally un­der­stands about mil­i­tary the­ory, be­cause it is an el­e­men­tary propo­si­tion that an un­co­or­di­nated mob gets slaugh­tered.

Sup­pose that a coun­try of ra­tio­nal­ists is at­tacked by a coun­try of Evil Bar­bar­ians who know noth­ing of prob­a­bil­ity the­ory or de­ci­sion the­ory.

Now there’s a cer­tain view­point on “ra­tio­nal­ity” or “ra­tio­nal­ism” which would say some­thing like this:

“Ob­vi­ously, the ra­tio­nal­ists will lose. The Bar­bar­ians be­lieve in an af­ter­life where they’ll be re­warded for courage; so they’ll throw them­selves into bat­tle with­out hes­i­ta­tion or re­morse. Thanks to their af­fec­tive death spirals around their Cause and Great Leader Bob, their war­riors will obey or­ders, and their cit­i­zens at home will pro­duce en­thu­si­as­ti­cally and at full ca­pac­ity for the war; any­one caught skim­ming or hold­ing back will be burned at the stake in ac­cor­dance with Bar­bar­ian tra­di­tion. They’ll be­lieve in each other’s good­ness and hate the en­emy more strongly than any sane per­son would, bind­ing them­selves into a tight group. Mean­while, the ra­tio­nal­ists will re­al­ize that there’s no con­ceiv­able re­ward to be had from dy­ing in bat­tle; they’ll wish that oth­ers would fight, but not want to fight them­selves. Even if they can find sol­diers, their civili­ans won’t be as co­op­er­a­tive: So long as any one sausage al­most cer­tainly doesn’t lead to the col­lapse of the war effort, they’ll want to keep that sausage for them­selves, and so not con­tribute as much as they could. No mat­ter how re­fined, el­e­gant, civ­i­lized, pro­duc­tive, and non­vi­o­lent their cul­ture was to start with, they won’t be able to re­sist the Bar­bar­ian in­va­sion; sane dis­cus­sion is no match for a froth­ing lu­natic armed with a gun. In the end, the Bar­bar­ians will win be­cause they want to fight, they want to hurt the ra­tio­nal­ists, they want to con­quer and their whole so­ciety is united around con­quest; they care about that more than any sane per­son would.”

War is not fun. As many many peo­ple have found since the dawn of recorded his­tory, as many many peo­ple have found out be­fore the dawn of recorded his­tory, as some com­mu­nity some­where is find­ing out right now in some sad lit­tle coun­try whose in­ter­nal ag­o­nies don’t even make the front pages any more.

War is not fun. Los­ing a war is even less fun. And it was said since the an­cient times: “If thou would have peace, pre­pare for war.” Your op­po­nents don’t have to be­lieve that you’ll win, that you’ll con­quer; but they have to be­lieve you’ll put up enough of a fight to make it not worth their while.

You per­ceive, then, that if it were gen­uinely the lot of “ra­tio­nal­ists” to always lose in war, that I could not in good con­science ad­vo­cate the wide­spread pub­lic adop­tion of “ra­tio­nal­ity”.

This is prob­a­bly the dirt­iest topic I’ve dis­cussed or plan to dis­cuss on LW. War is not clean. Cur­rent high-tech mil­i­taries—by this I mean the US mil­i­tary—are unique in the over­whelm­ingly su­pe­rior force they can bring to bear on op­po­nents, which al­lows for a his­tor­i­cally ex­traor­di­nary de­gree of con­cern about en­emy ca­su­alties and civilian ca­su­alties.

Win­ning in war has not always meant toss­ing aside all moral­ity. Wars have been won with­out us­ing tor­ture. The un­fun­ness of war does not im­ply, say, that ques­tion­ing the Pres­i­dent is un­pa­tri­otic. We’re used to “war” be­ing ex­ploited as an ex­cuse for bad be­hav­ior, be­cause in re­cent US his­tory that pretty much is ex­actly what it’s been used for...

But re­versed stu­pidity is not in­tel­li­gence. And re­versed evil is not in­tel­li­gence ei­ther. It re­mains true that real wars can­not be won by re­fined po­lite­ness. If “ra­tio­nal­ists” can’t pre­pare them­selves for that men­tal shock, the Bar­bar­ians re­ally will win; and the “ra­tio­nal­ists”… I don’t want to say, “de­serve to lose”. But they will have failed that test of their so­ciety’s ex­is­tence.

Let me start by dis­pos­ing of the idea that, in prin­ci­ple, ideal ra­tio­nal agents can­not fight a war, be­cause each of them prefers be­ing a civilian to be­ing a sol­dier.

As has already been dis­cussed at some length, I one-box on New­comb’s Prob­lem.

Con­sis­tently, I do not be­lieve that if an elec­tion is set­tled by 100,000 to 99,998 votes, that all of the vot­ers were ir­ra­tional in ex­pend­ing effort to go to the pol­ling place be­cause “my stay­ing home would not have af­fected the out­come”. (Nor do I be­lieve that if the elec­tion came out 100,000 to 99,999, then 100,000 peo­ple were all, in­di­vi­d­u­ally, solely re­spon­si­ble for the out­come.)

Con­sis­tently, I also hold that two ra­tio­nal AIs (that use my kind of de­ci­sion the­ory), even if they had com­pletely differ­ent util­ity func­tions and were de­signed by differ­ent cre­ators, will co­op­er­ate on the true Pri­soner’s Dilemma if they have com­mon knowl­edge of each other’s source code. (Or even just com­mon knowl­edge of each other’s ra­tio­nal­ity in the ap­pro­pri­ate sense.)

Con­sis­tently, I be­lieve that ra­tio­nal agents are ca­pa­ble of co­or­di­nat­ing on group pro­jects when­ever the (ex­pected prob­a­bil­is­tic) out­come is bet­ter than it would be with­out such co­or­di­na­tion. A so­ciety of agents that use my kind of de­ci­sion the­ory, and have com­mon knowl­edge of this fact, will end up at Pareto op­tima in­stead of Nash equil­ibria. If all ra­tio­nal agents agree that they are bet­ter off fight­ing than sur­ren­der­ing, they will fight the Bar­bar­ians rather than sur­ren­der.

Imag­ine a com­mu­nity of self-mod­ify­ing AIs who col­lec­tively pre­fer fight­ing to sur­ren­der, but in­di­vi­d­u­ally pre­fer be­ing a civilian to fight­ing. One solu­tion is to run a lot­tery, un­pre­dictable to any agent, to se­lect war­riors. Be­fore the lot­tery is run, all the AIs change their code, in ad­vance, so that if se­lected they will fight as a war­rior in the most com­mu­nally effi­cient pos­si­ble way—even if it means calmly march­ing into their own death.

(A re­flec­tively con­sis­tent de­ci­sion the­ory works the same way, only with­out the self-mod­ifi­ca­tion.)

You re­ply: “But in the real, hu­man world, agents are not perfectly ra­tio­nal, nor do they have com­mon knowl­edge of each other’s source code. Co­op­er­a­tion in the Pri­soner’s Dilemma re­quires cer­tain con­di­tions ac­cord­ing to your de­ci­sion the­ory (which these mar­gins are too small to con­tain) and these con­di­tions are not met in real life.”

I re­ply: The pure, true Pri­soner’s Dilemma is in­cred­ibly rare in real life. In real life you usu­ally have knock-on effects—what you do af­fects your rep­u­ta­tion. In real life most peo­ple care to some de­gree about what hap­pens to other peo­ple. And in real life you have an op­por­tu­nity to set up in­cen­tive mechanisms.

And in real life, I do think that a com­mu­nity of hu­man ra­tio­nal­ists could man­age to pro­duce sol­diers will­ing to die to defend the com­mu­nity. So long as chil­dren aren’t told in school that ideal ra­tio­nal­ists are sup­posed to defect against each other in the Pri­soner’s Dilemma. Let it be widely be­lieved—and I do be­lieve it, for ex­actly the same rea­son I one-box on New­comb’s Prob­lem—that if peo­ple de­cided as in­di­vi­d­u­als not to be sol­diers or if sol­diers de­cided to run away, then that is the same as de­cid­ing for the Bar­bar­ians to win. By that same the­ory whereby, if a lot­tery is won by 100,000 votes to 99,998 votes, it does not make sense for ev­ery voter to say “my vote made no differ­ence”. Let it be said (for it is true) that util­ity func­tions don’t need to be solip­sis­tic, and that a ra­tio­nal agent can fight to the death if they care enough about what they’re pro­tect­ing. Let them not be told that ra­tio­nal­ists should ex­pect to lose rea­son­ably.

If this is the cul­ture and the mores of the ra­tio­nal­ist so­ciety, then, I think, or­di­nary hu­man be­ings in that so­ciety would vol­un­teer to be sol­diers. That also seems to be built into hu­man be­ings, af­ter all. You only need to en­sure that the cul­tural train­ing does not get in the way.

And if I’m wrong, and that doesn’t get you enough vol­un­teers?

Then so long as peo­ple still pre­fer, on the whole, fight­ing to sur­ren­der; they have an op­por­tu­nity to set up in­cen­tive mechanisms, and avert the True Pri­soner’s Dilemma.

You can have lot­ter­ies for who gets elected as a war­rior. Sort of like the ex­am­ple above with AIs chang­ing their own code. Ex­cept that if “be re­flec­tively con­sis­tent; do that which you would pre­com­mit to do” is not suffi­cient mo­ti­va­tion for hu­mans to obey the lot­tery, then...

...well, in ad­vance of the lot­tery ac­tu­ally run­ning, we can per­haps all agree that it is a good idea to give the se­lectees drugs that will in­duce ex­tra courage, and shoot them if they run away. Even con­sid­er­ing that we our­selves might be se­lected in the lot­tery. Be­cause in ad­vance of the lot­tery, this is the gen­eral policy that gives us the high­est ex­pec­ta­tion of sur­vival. I said: Real wars = not fun, los­ing wars = less fun.

Let’s be clear, by the way, that I’m not en­dors­ing the draft as prac­ticed nowa­days. Those drafts are not col­lec­tive at­tempts by a pop­u­lace to move from a Nash equil­ibrium to a Pareto op­ti­mum. Drafts are a tool of kings play­ing games in need of toy sol­diers. The Viet­nam draf­tees who fled to Canada, I hold to have been in the right. But a so­ciety that con­sid­ers it­self too smart for kings, does not have to be too smart to sur­vive. Even if the Bar­bar­ian hordes are in­vad­ing, and the Bar­bar­ians do prac­tice the draft.

Will ra­tio­nal sol­diers obey or­ders? What if the com­mand­ing officer makes a mis­take?

Soldiers march. Every­one’s feet hit­ting the ground in the same rhythm. Even, per­haps, against their own in­cli­na­tions, since peo­ple left to them­selves would walk all at sep­a­rate paces. Lasers made out of peo­ple. That’s march­ing.

If it’s pos­si­ble to in­vent some method of group de­ci­sion­mak­ing that is su­pe­rior to the cap­tain hand­ing down or­ders, then a com­pany of ra­tio­nal sol­diers might im­ple­ment that pro­ce­dure. If there is no proven method bet­ter than a cap­tain, then a com­pany of ra­tio­nal sol­diers com­mit to obey the cap­tain, even against their own sep­a­rate in­cli­na­tions. And if hu­man be­ings aren’t that ra­tio­nal… then in ad­vance of the lot­tery, the gen­eral policy that gives you the high­est per­sonal ex­pec­ta­tion of sur­vival is to shoot sol­diers who di­s­obey or­ders. This is not to say that those who fragged their own officers in Viet­nam were in the wrong; for they could have con­sis­tently held that they preferred no one to par­ti­ci­pate in the draft lot­tery.

But an un­co­or­di­nated mob gets slaugh­tered, and so the sol­diers need some way of all do­ing the same thing at the same time in the pur­suit of the same goal, even though, left to their own de­vices, they might march off in all di­rec­tions. The or­ders may not come from a cap­tain like a su­pe­rior tribal chief, but unified or­ders have to come from some­where. A so­ciety whose sol­diers are too clever to obey or­ders, is a so­ciety which is too clever to sur­vive. Just like a so­ciety whose peo­ple are too clever to be sol­diers. That is why I say “clever”, which I of­ten use as a term of op­pro­brium, rather than “ra­tio­nal”.

(Though I do think it’s an im­por­tant ques­tion as to whether you can come up with a small-group co­or­di­na­tion method that re­ally gen­uinely in prac­tice works bet­ter than hav­ing a leader. The more peo­ple can trust the group de­ci­sion method—the more they can be­lieve that it re­ally is su­pe­rior to peo­ple go­ing their own way—the more co­her­ently they can be­have even in the ab­sence of en­force­able penalties for di­s­obe­di­ence.)

I say all this, even though I cer­tainly don’t ex­pect ra­tio­nal­ists to take over a coun­try any time soon, be­cause I think that what we be­lieve about a so­ciety of “peo­ple like us” has some re­flec­tion on what we think of our­selves. If you be­lieve that a so­ciety of peo­ple like you would be too rea­son­able to sur­vive in the long run… that’s one sort of self-image. And it’s a differ­ent sort of self-image if you think that a so­ciety of peo­ple all like you could fight the vi­cious Evil Bar­bar­ians and win—not just by dint of su­pe­rior tech­nol­ogy, but be­cause your peo­ple care about each other and about their col­lec­tive so­ciety—and be­cause they can face the re­al­ities of war with­out los­ing them­selves—and be­cause they would calcu­late the group-ra­tio­nal thing to do and make sure it got done—and be­cause there’s noth­ing in the rules of prob­a­bil­ity the­ory or de­ci­sion the­ory that says you can’t sac­ri­fice your­self for a cause—and be­cause if you re­ally are smarter than the Enemy and not just flat­ter­ing your­self about that, then you should be able to ex­ploit the blind spots that the Enemy does not al­low it­self to think about—and be­cause no mat­ter how heav­ily the Enemy hy­pes it­self up be­fore bat­tle, you think that just maybe a co­her­ent mind, un­di­vided within it­self, and per­haps prac­tic­ing some­thing akin to med­i­ta­tion or self-hyp­no­sis, can fight as hard in prac­tice as some­one who the­o­ret­i­cally be­lieves they’ve got sev­enty-two vir­gins wait­ing for them.

Then you’ll ex­pect more of your­self and peo­ple like you op­er­at­ing in groups; and then you can see your­self as some­thing more than a cul­tural dead end.

So look at it this way: Jeffreys­sai prob­a­bly wouldn’t give up against the Evil Bar­bar­ians if he were fight­ing alone. A whole army of beisut­sukai mas­ters ought to be a force that no one would mess with. That’s the mo­ti­vat­ing vi­sion. The ques­tion is how, ex­actly, that works.