How do humans assign utilities to world states?

It seems like a good por­tion of the whole “max­i­miz­ing util­ity” strat­egy which might be used by a sovereign re­lies on ac­tu­ally be­ing able to con­soli­date hu­man prefer­ences into util­ities. I think there are a few stages here, each of which may pre­sent ob­sta­cles. I’m not sure what the cur­rent state of the art is with re­gard to over­com­ing these, and am cu­ri­ous re­gard­ing such.

First, here are a few as­sump­tions that I’m us­ing just to make the prob­lem a bit more nav­i­gable (deal­ing with one or two hard prob­lems in­stead of a bunch at once) - will need to go back and do away with each of these (and each com­bi­na­tion thereof) and see what ad­di­tional prob­lems re­sult.

  1. The sovereign has in­finite com­put­ing power (and to shorten the list of as­sump­tions, can do 2-6 be­low)

  2. We’re max­i­miz­ing across the prefer­ences of a sin­gle hu­man (Alice for con­ve­nience). To the ex­tent that Alice cares about oth­ers, we’re ac­count­ing for their prefer­ences, too. But we’re not deal­ing with ag­gre­gat­ing prefer­ences across differ­ent sen­tient be­ings, yet. I think this is a sep­a­rate hard prob­lem.

  3. Alice has in­finite com­put­ing power.

  4. We’re as­sum­ing that Alice’s prefer­ences do not change and can­not change, ever, no mat­ter what hap­pens. So as Alice ex­pe­riences differ­ent things in her life, she has the ex­act same prefer­ences. No mat­ter what she learns or con­cludes about the world, she has the ex­act same prefer­ences. To be ex­plicit, this in­cludes prefer­ences re­gard­ing the rel­a­tive weight­ings of pre­sent and fu­ture wor­ld­states. (And in CEV terms, no spread, no dis­tance.)

  5. We’re as­sum­ing that Alice (and the sovereign) can de­duc­tively con­clude the fu­ture from the pre­sent, given a par­tic­u­lar course of ac­tion by the sovereign. Pic­ture a sin­gle his­tory of the uni­verse from the be­gin­ning of the uni­verse to now, and a bunch of wor­ldlines run­ning into the fu­ture de­pend­ing on what ac­tion the sovereign takes. To clar­ify, if you ask Alice about any sin­gle lit­tle de­tail across any of the fu­ture wor­ldlines, she can tell you that de­tail.

  6. Alice can read minds and the prefer­ences of other hu­mans and sen­tient be­ings (im­plied by 5, but try­ing to be ex­plicit.)

So Alice can con­clude any­thing and ev­ery­thing, pretty much (and so can our sovereign.) The sovereign is faced with the prob­lem of figur­ing out what ac­tion to take to max­i­mize across Alice’s prefer­ences. How­ever, Alice is ba­si­cally a sack of meat that has cer­tain emo­tions in re­sponse to cer­tain ex­pe­riences or cer­tain con­clu­sions about the world, and it doesn’t seem ob­vi­ous how to get the prefer­ence or­der­ing of the differ­ent wor­ldlines out of these emo­tions. Some difficul­ties:

  1. The sovereign no­tices that Alice ex­pe­riences differ­ent feel­ings in re­sponse to differ­ent stim­uli. How does the sovereign de­ter­mine which types of feel­ings to max­i­mize, and which to min­i­mize? There are a bunch of ways to deal with this, but most of them seem to have a chance of er­ror (and the con­junc­tion of p(er­ror) across all the times that the sovereign will need to do this ap­proach 1). For ex­am­ple, could train off an ex­ist­ing data set, could have it simu­late other hu­mans with ac­cess to Alice’s feel­ings and cog­ni­tion and have a simu­lated com­mit­tee dis­cuss and reach a de­ci­sion on each one, etc etc. But all of these boot­strap off of the as­sumed abil­ity of hu­mans to de­ter­mine which feel­ings to max­i­mize (just with amped up com­put­ing power) - this doesn’t strike me as a satis­fac­tory solu­tion.

  2. As­sume 1. is solved. The sovereign knows which feel­ings to max­i­mize. How­ever, it’s ended up with a bunch of axes. How does it de­ter­mine the ap­pro­pri­ate trade-offs to make? (Or, to put it an­other way, how does it de­ter­mine the rel­a­tive value of differ­ent po­si­tions along each axis with differ­ent po­si­tions along differ­ent axes?)

So, to re­hash my ac­tual re­quest: what’s the state of the art with re­gards to these difficul­ties, and how con­fi­dent are we that we’ve reached a satis­fac­tory an­swer?