# Wei_Dai comments on Metaphilosophical Mysteries

• Part of what con­fuses me is that you said we’re as­sign­ing plau­si­bil­ity to classes of struc­tures, not in­di­vi­d­ual struc­tures, but it seems like we’d need to as­sign plau­si­bil­ity to in­di­vi­d­ual struc­tures in prac­tice.

How to use this in de­ci­sion-mak­ing is a spe­cial case of a more gen­eral open prob­lem in am­bi­ent con­trol.

Can’t you give an ex­am­ple us­ing a situ­a­tion where Bayesian up­dat­ing is non-prob­le­matic, and just show how we might use your idea for the prior with stan­dard de­ci­sion the­ory?

• If you can re­fer to an in­di­vi­d­ual struc­ture in­for­mally, then ei­ther there is a lan­guage that al­lows finitely de­scribing it, or abil­ity to re­fer to that struc­ture is an illu­sion and in fact you are only refer­ring to some big­ger col­lec­tion of things (prop­erty) of which the ob­ject you talk about is an el­e­ment. If you can’t re­fer to a struc­ture, then you don’t need plau­si­bil­ity for it.

Can’t you give an ex­am­ple us­ing a situ­a­tion where Bayesian up­dat­ing is non-prob­le­matic, and just show how we might use your idea for the prior with stan­dard de­ci­sion the­ory?

This is only helpful is some­thing works with tricky math­e­mat­i­cal struc­tures, and in all cases that seems to need to be prefer­ence. For ex­am­ple, you’d pre­fer to make de­ci­sions that are (likely!) con­sis­tent with a given the­ory (make it hold), then it helps if your de­ci­sion and that the­ory are ex­pressed in the same set­ting (lan­guage), and you can make de­ci­sions un­der log­i­cal un­cer­tainty if you use the uni­ver­sal prior on state­ments. Nor­mally, de­ci­sion the­o­ries don’t con­sider such cases, so I’m not sure how to re­late. In­tro­duc­ing ob­ser­va­tions will prob­a­bly be a mis­take too.

• ei­ther there is a lan­guage that al­lows finitely de­scribing it

But if you fix a lan­guage L for your uni­ver­sal prior, then there will be a more pow­er­ful met­a­lan­guage L’ that al­lows finitely de­scribing some struc­ture, which can’t be finitely de­scribed in the base lan­guage, right? So don’t we still have the prob­lem of the uni­ver­sal prior not re­ally be­ing uni­ver­sal?

I can’t parse the sec­ond part of your re­sponse. Will keep try­ing...

• But if you fix a lan­guage L for your uni­ver­sal prior, then there will be a more pow­er­ful met­a­lan­guage L’ that al­lows finitely de­scribing some struc­ture, which can’t be finitely de­scribed in the base lan­guage, right? So don’t we still have the prob­lem of the uni­ver­sal prior not re­ally be­ing uni­ver­sal?

It can still talk about all struc­tures, but some­times won’t be able to point at a spe­cific struc­ture, only a class con­tain­ing it. You only need a lan­guage ex­pres­sive enough to de­scribe ev­ery­thing prefer­ence refers to, and no more. (This seems to be the cor­rect solu­tion to on­tol­ogy prob­lem—de­scribe prefer­ence as be­ing about math­e­mat­i­cal struc­tures (more gen­er­ally, con­cepts/​the­o­ries), and ig­nore the ques­tion of the na­ture of re­al­ity.)

(Clar­ified the sec­ond part of the pre­vi­ous com­ment a bit.)

• You only need a lan­guage ex­pres­sive enough to de­scribe ev­ery­thing prefer­ence refers to, and no more.

Why do you think that any log­i­cal lan­guage (of the sort we’re cur­rently fa­mil­iar with) is suffi­ciently ex­pres­sive for this pur­pose?

This seems to be the cor­rect solu­tion to on­tol­ogy prob­lem—de­scribe prefer­ence as be­ing about math­e­mat­i­cal struc­tures (more gen­er­ally, con­cepts/​the­o­ries), and ig­nore the ques­tion of the na­ture of re­al­ity.

I’m not sure. One way to think about it is whether the ques­tion “what is the right prior?” is more like “what is the right de­ci­sion the­ory?” or more like “what is the right util­ity func­tion?” In What Are Prob­a­bil­ities, Any­way? I es­sen­tially said that I lean to­wards the lat­ter, but I’m highly un­cer­tain.

ETA: And some­times I sus­pect even “what is the right util­ity func­tion?” is re­ally more like “what is the right de­ci­sion the­ory?” than we cur­rently be­lieve. In other words there is ob­jec­tive moral­ity af­ter all, but we’re cur­rently just too stupid or philo­soph­i­cally in­com­pe­tent to figure out what it is.

• Why do you think that any log­i­cal lan­guage (of the sort we’re cur­rently fa­mil­iar with) is suffi­ciently ex­pres­sive for this pur­pose?

The gen­eral idea seems right. If the ex­ist­ing lan­guages are in­ad­e­quate, they at least seem ad­e­quate for a full-fea­tured pro­to­type: figure out de­ci­sion the­ory (and hence no­tion of prefer­ence) in terms of stan­dard logic, then move on as nec­es­sary for ex­tend­ing ex­pres­sive power. This should stop at some point, since this ex­er­cise at for­mal­ity is aimed at con­struc­tion of a pro­gram.

I’m not sure. One way to think about it is whether the ques­tion “what is the right prior?” is more like “what is the right de­ci­sion the­ory?” or more like “what is the right util­ity func­tion?” In What Are Prob­a­bil­ities, Any­way? I es­sen­tially said that I lean to­wards the lat­ter, but I’m highly un­cer­tain.

I don’t see clearly the dis­tinc­tion you’re mak­ing, so let me de­scribe how I see it. Some de­sign choices in con­struct­ing FAI would cer­tainly be spe­cific to our minds (val­ues), but the main as­sump­tion to my ap­proach to FAI is ex­actly that a large por­tion of de­sign choices in FAI can be speci­fied as a nat­u­ral cat­e­gory in hu­man brains, some­thing we can point a sim­ple mir­ror at and say “there!”, with the mir­ror do­ing most of the work in de­ter­min­ing what goes into the FAI. I call the au­to­mated de­sign choices “prefer­ence”, and the mir­ror (the­ory of mir­ror) “de­ci­sion the­ory”, with the slot “no­tion of prefer­ence” that is to be filled in au­to­mat­i­cally. So, there is no ques­tion of which one of “de­ci­sion the­ory” and “prefer­ence” is “es­sen­tial”, both play a role. The worry is about the nec­es­sary size of the man­u­ally de­signed “de­ci­sion the­ory” part, and whether it’s hu­manly pos­si­ble to con­struct it.