Common mistakes people make when thinking about decision theory

From my ex­pe­rience read­ing and talk­ing about de­ci­sion the­ory on LW, it seems that many of the un­pro­duc­tive com­ments in these dis­cus­sions can be at­tributed to a hand­ful of com­mon mis­takes.

Mis­take #1: Ar­gu­ing about assumptions

The main rea­son why I took so long to un­der­stand New­comb’s Prob­lem and Coun­ter­fac­tual Mug­ging was my in­sis­tence on deny­ing the as­sump­tions be­hind these puz­zles. I could have saved months if I’d just said to my­self, okay, is this di­rec­tion of in­quiry in­ter­est­ing when taken on its own terms?

Many as­sump­tions seemed to be di­vorced from real life at first. Peo­ple dis­missed the study of elec­tro­mag­netism as an im­prac­ti­cal toy, and con­sid­ered num­ber the­ory hope­lessly ab­stract un­til cryp­tog­ra­phy ar­rived. The only way to make in­tel­lec­tual progress (ei­ther in­di­vi­d­u­ally or as a group) is to ex­plore the im­pli­ca­tions of in­ter­est­ing as­sump­tions wher­ever they might lead. Un­for­tu­nately peo­ple love to ar­gue about as­sump­tions in­stead of get­ting any­thing done, though they can’t re­ally judge be­fore ex­plor­ing the im­pli­ca­tions in de­tail.

Sev­eral smart peo­ple on LW are re­peat­ing my ex­act mis­take about New­comb’s Prob­lem now, and oth­ers find ways to com­mit the same mis­take when look­ing at our newer ideas. It’s so frus­trat­ing and un­in­ter­est­ing to read yet an­other com­ment say­ing my as­sump­tions look un­in­tu­itive or un­phys­i­cal or ir­rele­vant to FAI or what­ever. I’m not against crit­i­cism, but some­how such com­ments never blos­som into in­ter­est­ing con­ver­sa­tions, and that’s rea­son enough to cau­tion you against the way of think­ing that causes them.

Mis­take #2: Stop­ping when your idea seems good enough

There’s a hand­ful of ideas that de­ci­sion the­ory new­bies re­dis­cover again and again, like point­ing out in­dex­i­cal un­cer­tainty as the solu­tion to New­comb’s prob­lem, or adding ran­dom­ness to mod­els of UDT to elimi­nate spu­ri­ous proofs. Th­ese ideas don’t work and don’t lead any­where in­ter­est­ing, but that’s hard to no­tice when you just had the flash of in­sight and want to share it with the world.

A good strat­egy in such situ­a­tions is to always push a lit­tle bit past the point where you have ev­ery­thing figured out. Take one ex­tra step and ask your­self: “Can I make this idea pre­cise?” What are the first few im­pli­ca­tions? What are the ob­vi­ous ex­ten­sions? If your re­sult seems to con­tra­dict what’s already known, work through some of the con­tra­dic­tions your­self. If you don’t find any mis­takes in your idea, you will surely find new for­mal things to say about your idea, which always helps.

Mis­take #2A: Stop­ping when your idea ac­tu­ally is good enough

I didn’t want to name any names in this post be­cause my sta­tus on LW puts me in a kinda po­si­tion of power, but there’s a name I can name with a clear con­science. In 2009, Eliezer wrote:

For­mally you’d use a Godelian di­ag­o­nal to write (...)

Of course that’s not a new­bie mis­take at all, but an awe­some and fruit­ful idea! As it hap­pens, writ­ing out that Godelian di­ag­o­nal im­me­di­ately leads to all sorts of puz­zling ques­tions like “but what does it ac­tu­ally do? and how do we prove it?”, and even­tu­ally to all the de­ci­sion the­ory re­search we’re do­ing now. Know­ing Eliezer’s in­tel­li­gence, he prob­a­bly could have pre­empted most of our re­sults. In­stead he just de­clared the prob­lem solved. Maybe he thought he was already at 0.95 for­mal­ity and that go­ing to 1.0 would be a triv­ial step? I don’t want to in­sinu­ate here, but IMO he made a mis­take.

Since this mis­take is in­dis­t­in­guish­able from the last, the rem­edy for it is the same: “Can I make this idea pre­cise?” When­ever you stake out a small area of knowl­edge and make it amenable to math­e­mat­i­cal think­ing, you’re likely to find new math that has last­ing value. When you stop be­cause your not-quite-for­mal idea seems already good enough, you squan­der that op­por­tu­nity.


If this post has con­vinced you to stop mak­ing these com­mon mis­takes, be warned that it won’t nec­es­sar­ily make you hap­pier. As you learn to see more clearly, the first thing you’ll see will be a locked door with a sign say­ing “Re­search is hard”. Though it’s not very scary or heroic, mostly you just stand there feel­ing stupid about your­self :-)