Strong implication of preference uncertainty

Here is a the­ory that is just as good as gen­eral rel­a­tivity:

AGR (An­gel Gen­eral Rel­a­tivity): Tiny in­visi­ble an­gels push around all the par­ti­cles in the uni­verse in a way that is in­dis­t­in­guish­able from the equa­tions of gen­eral rel­a­tivity.

This the­ory is falsifi­able, just as gen­eral rel­a­tivity (GR) it­self is. In­deed, since it gives ex­actly the same pre­dic­tions as GR, a Bayesian will never find ev­i­dence that prefers it over Ein­stein’s the­ory.

There­fore, I ob­vi­ously de­serve a No­bel prize for sug­gest­ing it.

En­ter Oc­cam’s shav­ing equipment

Ob­vi­ously the an­gel the­ory is not a rev­olu­tion­ary new the­ory. Par­tially be­cause I’ve not done any of the hard work, just con­structed a poin­ter to Ein­stein’s the­ory. But, philo­soph­i­cally, the main jus­tifi­ca­tion is Oc­cam’s ra­zor—the sim­plest the­ory is to be preferred.

From a Bayesian per­spec­tive, you could see vi­o­la­tions of Oc­cam’s ra­zor as cheat­ing, us­ing your pos­te­rior as pri­ors. There is a whole class of “an­gels are push­ing par­ti­cles” the­o­ries, and AGR is just a small por­tion of that space. By con­sid­er­ing AGR and GR on equal foot­ing, we’re priv­ileg­ing AGR above what it de­serves[1].

In physics, Oc­cam’s ra­zor doesn’t mat­ter for strictly iden­ti­cal theories

Oc­cam’s ra­zor has two roles: the first is to dis­t­in­guish be­tween strictly iden­ti­cal the­o­ries; the sec­ond is to dis­t­in­guish be­tween the­o­ries that give the same pre­dic­tion on the data so far, but may differ in the fu­ture.

Here, we fo­cus on the first case: GR and AGR are strictly iden­ti­cal; no data will ever dis­t­in­guish them. In essence, the the­ory that one is right and the other wrong is not falsifi­able.

What that means is that, though AGR may be a pri­ori less likely than GR, the rel­a­tive prob­a­bil­ity be­tween the two the­o­ries will never change: they make the same pre­dic­tions. And also be­cause they make the same pre­dic­tions, that rel­a­tive prob­a­bil­ity is ir­rele­vant in prac­tice: we could use AGR just as well as GR for pre­dic­tions.

How prefer­ences differ

Now let’s turn to prefer­ences, as de­scribed in our pa­per “Oc­cam’s ra­zor is in­suffi­cient to in­fer the prefer­ences of ir­ra­tional agents”.

Here two sets of prefer­ences are “pre­dic­tion-iden­ti­cal”, in the sense of the physics the­o­ries above, if they pre­dict the same be­havi­our for the agent. So that means that two differ­ent prefer­ence-based ex­pla­na­tions for the same be­havi­our will never change their rel­a­tive prob­a­bil­ities.

Worse than that, Oc­cam’s ra­zor doesn’t solve the is­sue. The sim­plest ex­pla­na­tions of, say, hu­man be­havi­our, is that hu­mans are fully ra­tio­nal at all times. This isn’t the ex­pla­na­tion that we want.

Even worse than that, pre­dic­tion-iden­ti­cal prefer­ences will lead to vastly differ­ent con­se­quences if pro­gram an AI to max­imise them.

So, in sum­mary:

  1. Pre­dic­tion-iden­ti­cal prefer­ences never change rel­a­tive prob­a­bil­ity.

  2. The sim­plest pre­dic­tion-iden­ti­cal prefer­ences are known to be wrong for hu­mans.

  3. It could be very im­por­tant for the fu­ture to get the right prefer­ence for hu­mans.

  1. GR would make up a larger por­tion of , “ge­o­met­ric the­o­ries of space-time” than AGR makes up of , and would be more likely than any­way, es­pe­cially af­ter up­dat­ing on the non-ob­ser­va­tion of an­gels. ↩︎