# Eliezer Yudkowsky comments on Towards a New Decision Theory

• Why didn’t you men­tion ear­lier that your time­less de­ci­sion the­ory mainly had to do with log­i­cal un­cer­tainty?

Be­cause I was think­ing in terms of sav­ing it for a PhD the­sis or some other pub­li­ca­tion, and if you get that in­sight the rest fol­lows pretty fast—did for me at least. Also I was us­ing it as a test for would-be AI re­searchers: “Here’s New­comblike prob­lems, here’s why the clas­si­cal solu­tion doesn’t work for self-mod­ify­ing AI, can you solve this FAI prob­lem which I know to be solv­able?”

I still think (B) is true, BTW. We should de­vote some time and re­sources to think­ing about how we are solv­ing these prob­lems (and com­ing up with ques­tions in the first place). Find­ing that al­gorithm is per­haps more im­por­tant than find­ing a re­flec­tively con­sis­tent de­ci­sion al­gorithm, if we don’t want an AI to be stuck with what­ever mis­takes we might make.

And yet you found a re­flec­tively con­sis­tent de­ci­sion al­gorithm long be­fore you found a de­ci­sion-sys­tem-al­gorithm-find­ing al­gorithm. That’s not co­in­ci­dence. The lat­ter prob­lem is much harder. I sus­pect that even an in­for­mal un­der­stand­ing of parts of it would mean that you could find time­less de­ci­sion the­ory as eas­ily as fal­ling back­ward off a tree—you just run the al­gorithm in your own head. So with vey high prob­a­bil­ity you are go­ing to start see­ing through the ob­ject-level prob­lems be­fore you see through the meta ones. Con­versely I am EXTREMELY skep­ti­cal of peo­ple who claim they have an al­gorithm to solve meta prob­lems but who still seem con­fused about ob­ject prob­lems. Take metaethics, a solved prob­lem: what are the odds that some­one who still thought metaethics was a Deep Mys­tery could write an AI al­gorithm that could come up with a cor­rect metaethics? I tried that, you know, and in ret­ro­spect it didn’t work.

The meta al­gorithms are im­por­tant but by their very na­ture, know­ing even a lit­tle about the meta-prob­lem tends to make the ob­ject prob­lem much less con­fus­ing, and you will progress on the ob­ject prob­lem faster than on the meta prob­lem. Again, that’s not say­ing the meta prob­lem is im­por­tant. It’s just say­ing that it’s re­ally hard to end up in a state where meta has re­ally truly run ahead of ob­ject, though it’s easy to get illu­sions of hav­ing done so.

• It’s in­ter­est­ing that we came upon the same idea from differ­ent di­rec­tions. For me it fell out of Teg­mark’s mul­ti­verse. What could con­se­quences be, ex­cept log­i­cal con­se­quences, if all math­e­mat­i­cal struc­tures ex­ist? The fact that you said it would take a long se­ries of posts to ex­plain your idea threw me off, and I was kind of sur­prised when you said con­grat­u­la­tions. I thought I might be offer­ing a differ­ent solu­tion. (I spent days pol­ish­ing the ar­ti­cle in the ex­pec­ta­tion that I might have to defend it fiercely.)

And yet you found a re­flec­tively con­sis­tent de­ci­sion al­gorithm long be­fore you found a de­ci­sion-sys­tem-al­gorithm-find­ing al­gorithm. That’s not co­in­ci­dence. The lat­ter prob­lem is much harder.

Umm, I haven’t ac­tu­ally found a re­flec­tively con­sis­tent de­ci­sion al­gorithm yet, since the pro­posal has huge gaps that need to be filled. I have lit­tle idea how to han­dle log­i­cal un­cer­tainty in a sys­tem­atic way, or whether ex­pected util­ity max­i­miza­tion makes sense in that con­text.

The rest of your para­graph makes good points. But I’m not sure what you mean by “metaethics, a solved prob­lem”. Can you give a link?

One way to ap­proach the meta prob­lem may be to con­sider the meta-meta prob­lem: why did evolu­tion cre­ate us with so much “com­mon sense” on these types of prob­lems? Why do we have the meta al­gorithm ap­par­ently “built in” when it doesn’t seem like it would have offered much ad­van­tage in the an­ces­tral en­vi­ron­ment?

• But I’m not sure what you mean by “metaethics, a solved prob­lem”. Can you give a link?

http://​​wiki.less­wrong.com/​​wiki/​​Me­taethics_sequence

(Ob­serve that this page was cre­ated af­ter you asked the ques­tion. And I’m quite aware that it needs a bet­ter sum­mary—maybe “A Nat­u­ral Ex­pla­na­tion of Me­taethics” or the like.)

• The fact that you said it would take a long se­ries of posts to ex­plain your idea threw me off, and I was kind of sur­prised when you said congratulations

“De­cide as though your de­ci­sion is about the out­put of a Pla­tonic com­pu­ta­tion” is the key in­sight that started me off—not the only idea—and con­sid­er­ing how long philoso­phers have wran­gled this, there’s the whole ed­ifice of jus­tifi­ca­tion that would be needed for a se­ri­ous ex­po­si­tion. Maybe come Aug 26th or there­abouts I’ll post a very quick sum­mary of e.g. in­te­gra­tion with Pearl’s causal­ity.

• The usual rea­son for build­ing things in is that it re­duces trial-and-er­ror learn­ing. That’s good if the er­rors are ex­pen­sive and have a nega­tive im­pact on fit­ness.

Is there some­thing wrong with that ex­pla­na­tion in this con­text?