Contaminated by Optimism

Fol­lowup to: An­thro­po­mor­phic Op­ti­mism, The Hid­den Com­plex­ity of Wishes

Yes­ter­day, I reprised in fur­ther de­tail The Tragedy of Group Selec­tion­ism, in which early biol­o­gists be­lieved that preda­tors would vol­un­tar­ily re­strain their breed­ing to avoid ex­haust­ing the prey pop­u­la­tion; the given ex­cuse was “group se­lec­tion”. Not only does it turn out to be nearly im­pos­si­ble for group se­lec­tion to over­come a coun­ter­vailing in­di­vi­d­ual ad­van­tage; but when these nigh-im­pos­si­ble con­di­tions were cre­ated in the lab­o­ra­tory—group se­lec­tion for low-pop­u­la­tion groups—the ac­tual re­sult was not re­straint in breed­ing, but, of course, can­ni­bal­ism, es­pe­cially of im­ma­ture fe­males.

I’ve made even sillier mis­takes, by the way—though about AI, not evolu­tion­ary biol­ogy. And the thing that strikes me, look­ing over these cases of an­thro­po­mor­phism, is the ex­tent to which you are screwed as soon as you let an­thro­po­mor­phism sug­gest ideas to ex­am­ine.

In large hy­poth­e­sis spaces, the vast ma­jor­ity of the cog­ni­tive la­bor goes into notic­ing the true hy­poth­e­sis. By the time you have enough ev­i­dence to con­sider the cor­rect the­ory as one of just a few plau­si­ble al­ter­na­tives—to rep­re­sent the cor­rect the­ory in your mind—you’re prac­ti­cally done. Of this I have spo­ken sev­eral times be­fore.

And by the same to­ken, my ex­pe­rience sug­gests that as soon as you let an­thro­po­mor­phism pro­mote a hy­poth­e­sis to your at­ten­tion, so that you start won­der­ing if that par­tic­u­lar hy­poth­e­sis might be true, you’ve already com­mit­ted most of the mis­take.

The group se­lec­tion­ists did not de­liber­ately ex­tend credit to the be­lief that evolu­tion would do the aes­thetic thing, the nice thing. The group se­lec­tion­ists were doomed when they let their aes­thetic sense make a sug­ges­tion—when they let it pro­mote a hy­poth­e­sis to the level of de­liber­ate con­sid­er­a­tion.

It’s not like I knew the origi­nal group se­lec­tion­ists. But I’ve made analo­gous mis­takes as a teenager, and then watched oth­ers make the mis­take many times over. So I do have some ex­pe­rience whereof I speak, when I speak of in­stant doom.

Un­for­tu­nately, the pro­phy­lac­tic against this mis­take, is not a rec­og­nized tech­nique of Tra­di­tional Ra­tion­al­ity.

In Tra­di­tional Ra­tion­al­ity, you can get your ideas from any­where. Then you weigh up the ev­i­dence for and against them, search­ing for ar­gu­ments on both sides. If the ques­tion hasn’t been definitely set­tled by ex­per­i­ment, you should try to do an ex­per­i­ment to test your opinion, and du­tifully ac­cept the re­sult.

“Sorry, you’re not al­lowed to sug­gest ideas us­ing that method” is not some­thing you hear, un­der Tra­di­tional Ra­tion­al­ity.

But it is a fact of life, an ex­per­i­men­tal re­sult of cog­ni­tive psy­chol­ogy, that when peo­ple have an idea from any source, they tend to search for sup­port rather than con­tra­dic­tion—even in the ab­sence of emo­tional com­mit­ment (see link).

It is a fact of life that prim­ing and con­tam­i­na­tion oc­cur: just be­ing briefly ex­posed to com­pletely un­in­for­ma­tive, known false, or to­tally ir­rele­vant “in­for­ma­tion” can ex­ert sig­nifi­cant in­fluence on sub­jects’ es­ti­mates and de­ci­sions. This hap­pens on a level be­low de­liber­ate aware­ness, and that’s go­ing to be pretty hard to beat on prob­lems where an­thro­po­mor­phism is bound to rush in and make sug­ges­tions—but at least you can avoid de­liber­ately mak­ing it worse.

It is a fact of life that we change our minds less of­ten than we think. Once an idea gets into our heads, it is harder to get it out than we think. Only an ex­tremely re­stric­tive chain of rea­son­ing, that definitely pro­hibited most pos­si­bil­ities from con­sid­er­a­tion, would be suffi­cient to undo this dam­age—to root an idea out of your head once it lodges. The less you know for sure, the eas­ier it is to be­come con­tam­i­nated—weak do­main knowl­edge in­creases con­tam­i­na­tion effects.

It is a fact of life that we are far more likely to stop search­ing for fur­ther al­ter­na­tives at a point when we have a con­clu­sion we like, than when we have a con­clu­sion we dis­like.

It is a fact of life that we hold ideas we would like to be­lieve, to a lower stan­dard of proof than ideas we would like to dis­be­lieve. In the former case we ask “Am I al­lowed to be­lieve it?” and in the lat­ter case ask “Am I forced to be­lieve it?” If your do­main knowl­edge is weak, you will not know enough for your own knowl­edge to grab you by the throat and tell you “You’re wrong! That can’t pos­si­bly be true!” You will find that you are al­lowed to be­lieve it. You will search for plau­si­ble-sound­ing sce­nar­ios where your be­lief is true. If the search space of pos­si­bil­ities is large, you will al­most cer­tainly find some “win­ners”—your do­main knowl­edge be­ing too weak to definitely pro­hibit those sce­nar­ios.

It is a fact of his­tory that the group se­lec­tion­ists failed to re­lin­quish their folly. They found what they thought was a perfectly plau­si­ble way that evolu­tion (evolu­tion!) could end up pro­duc­ing foxes who vol­un­tar­ily avoided re­pro­duc­tive op­por­tu­ni­ties(!). And the group se­lec­tion­ists did in fact cling to that hy­poth­e­sis. That’s what hap­pens in real life! Be warned!

To beat an­thro­po­mor­phism you have to be scared of let­ting an­thro­po­mor­phism make sug­ges­tions. You have to try to avoid be­ing con­tam­i­nated by an­thro­po­mor­phism (to the best ex­tent you can).

As soon as you let an­thro­po­mor­phism gen­er­ate the idea and ask, “Could it be true?” then your brain has already swapped out of for­ward-ex­trap­o­la­tion mode and into back­ward-ra­tio­nal­iza­tion mode. Tra­di­tional Ra­tion­al­ity con­tains in­ad­e­quate warn­ings against this, IMO. See in par­tic­u­lar the post where I ar­gue against the Tra­di­tional in­ter­pre­ta­tion of Devil’s Ad­vo­cacy.

Yes, there are oc­ca­sions when you want to perform ab­duc­tive in­fer­ence, such as when you have ev­i­dence that some­thing is true and you are ask­ing how it could be true. We call that “Bayesian up­dat­ing”, in fact. An oc­ca­sion where you don’t have any ev­i­dence but your brain has made a cute lit­tle an­thro­po­mor­phic sug­ges­tion, is not a time to start won­der­ing how it could be true. Espe­cially if the search space of pos­si­bil­ities is large, and your do­main knowl­edge is too weak to pro­hibit plau­si­ble-sound­ing sce­nar­ios. Then your pre­dic­tion ends up be­ing de­ter­mined by an­thro­po­mor­phism. If the real pro­cess is not con­trol­led by a brain similar to yours, this is not a good thing for your pre­dic­tive ac­cu­racy.

This is a war I wage pri­mar­ily on the bat­tle­ground of Un­friendly AI, but it seems to me that many of the con­clu­sions ap­ply to op­ti­mism in gen­eral.

How did the idea first come to you, that the sub­prime melt­down wouldn’t de­crease the value of your in­vest­ment in Dan­ish deu­terium deriva­tives? Were you just think­ing neu­trally about the course of fi­nan­cial events, try­ing to ex­trap­o­late some of the many differ­ent ways that one fi­nan­cial billiard ball could ri­co­chet off an­other? Even this method tends to be sub­ject to op­ti­mism; if we know which way we want each step to go, we tend to vi­su­al­ize it go­ing that way. But bet­ter that, than start­ing with a pure hope—an out­come gen­er­ated be­cause it ranked high in your prefer­ence or­der­ing—and then per­mit­ting your mind to in­vent plau­si­ble-sound­ing rea­sons it might hap­pen. This is just rush­ing to failure.

And to spell out the ap­pli­ca­tion to Un­friendly AI: You’ve got var­i­ous peo­ple in­sist­ing that an ar­bi­trary mind, in­clud­ing an ex­pected pa­per­clip max­i­mizer, would do var­i­ous nice things or obey var­i­ous com­fort­ing con­di­tions: “Keep hu­mans around, be­cause di­ver­sity is im­por­tant to cre­ativity, and the hu­mans will provide a differ­ent point of view.” Now you might want to se­ri­ously ask if, even grant­ing that premise, you’d be kept in a nice house with air con­di­tion­ing; or kept in a tiny cell with life sup­port tubes and reg­u­lar elec­tric shocks if you didn’t gen­er­ate enough in­ter­est­ing ideas that day (and of course you wouldn’t be al­lowed to die); or up­loaded to a very small com­puter some­where, and restarted ev­ery cou­ple of years. No, let me guess, you’ll be more pro­duc­tive if you’re happy. So it’s clear why you want that to be the ar­gu­ment; but un­like you, the pa­per­clip max­i­mizer is not fran­ti­cally search­ing for a rea­son not to tor­ture you.

Sorry, the whole sce­nario is still around as un­likely as your care­fully pick­ing up ants on the side­walk, rather than step­ping on them, and keep­ing them in a happy ant colony for the sole ex­press pur­pose of sug­gest­ing blog com­ments. There are rea­sons in my goal sys­tem to keep sen­tient be­ings al­ive, even if they aren’t “use­ful” at the mo­ment. But from the per­spec­tive of a Bayesian su­per­in­tel­li­gence whose only ter­mi­nal value is pa­per­clips, it is not an op­ti­mal use of mat­ter and en­ergy to­ward the in­stru­men­tal value of pro­duc­ing di­verse and cre­ative ideas for mak­ing pa­per­clips, to keep around six billion highly similar hu­man brains. Un­like you, the pa­per­clip max­i­mizer doesn’t start out know­ing it wants that to be the con­clu­sion.

Your brain starts out know­ing that it wants hu­man­ity to live, and so it starts try­ing to come up with ar­gu­ments for why that is a perfectly rea­son­able thing for a pa­per­clip max­i­mizer to do. But the pa­per­clip max­i­mizer it­self would not start from the con­clu­sion that it wanted hu­man­ity to live, and rea­son back­ward. It would just try to make pa­per­clips. It wouldn’t stop, the way your own mind tends to stop, if it did find one ar­gu­ment for keep­ing hu­mans al­ive; in­stead it would go on search­ing for an even su­pe­rior al­ter­na­tive, some way to use the same re­sources to greater effect. Maybe you just want to keep 20 hu­mans and ran­domly per­turb their brain states a lot.

If you can’t blind your eyes to hu­man goals and just think about the pa­per­clips, you can’t un­der­stand what the goal of mak­ing pa­per­clips im­plies. It’s like ex­pect­ing kind and mer­ciful re­sults from nat­u­ral se­lec­tion, which lets old elephants starve to death when they run out of teeth.

A pri­ori, if you want a nice re­sult that takes 10 bits to spec­ify, then a pri­ori you should ex­pect a 1/​1024 prob­a­bil­ity of find­ing that some un­re­lated pro­cess gen­er­ates that nice re­sult. And a gen­uinely nice out­come in a large out­come space takes a lot more in­for­ma­tion than the English word “nice”, be­cause what we con­sider a good out­come has many com­po­nents of value. It’s ex­tremely sus­pi­cious if you start out with a nice re­sult in mind, search for a plau­si­ble rea­son that a not-in­her­ently-nice pro­cess would gen­er­ate it, and, by golly, find an amaz­ing clever ar­gu­ment.

And the more com­plex­ity you add to your re­quire­ments—hu­mans not only have to sur­vive, but have to sur­vive un­der what we would con­sider good liv­ing con­di­tions, etc. - the less you should ex­pect, a pri­ori, a non-nice pro­cess to gen­er­ate it. The less you should ex­pect to, amaz­ingly, find a gen­uine valid rea­son why the non-nice pro­cess hap­pens to do what you want. And the more sus­pi­cious you should be, if you find a clever-sound­ing ar­gu­ment why this should be the case. To ex­pect this to hap­pen with non-triv­ial prob­a­bil­ity is pul­ling in­for­ma­tion from nowhere; a blind ar­row is hit­ting the cen­ter of a small tar­get. Are you sure it’s wise to even search for such pos­si­bil­ities? Your chance of de­ceiv­ing your­self is far greater than the a pri­ori chance of a good out­come, es­pe­cially if your do­main knowl­edge is too weak to definitely rule out pos­si­bil­ities.

No more than you can guess a lot­tery ticket, should you ex­pect a pro­cess not shaped by hu­man nice­ness, to pro­duce nice re­sults in a large out­come space. You may not know the do­main very well, but you can un­der­stand that, a pri­ori, “nice” re­sults re­quire spe­cific com­plex­ity to hap­pen for no rea­son, and com­plex spe­cific mir­a­cles are rare.

I wish I could tell peo­ple: “Stop! Stop right there! You defeated your­self the mo­ment you knew what you wanted! You need to throw away your thoughts and start over with a neu­tral for­ward ex­trap­o­la­tion, not seek­ing any par­tic­u­lar out­come.” But the in­fer­en­tial dis­tance is too great; and then be­gins the slog of, “I don’t see why that couldn’t hap­pen” and “I don’t think you’ve proven my idea is wrong.”

It’s Un­friendly su­per­in­tel­li­gence that tends to worry me most, of course. But I do think the point gen­er­al­izes to quite a lot of op­ti­mism. You may know what you want, but Na­ture doesn’t care.