A Master-Slave Model of Human Preferences

[Wei Dai]

[This post is an ex­pan­sion of my pre­vi­ous open thread com­ment, and largely in­spired by Robin Han­son’s writ­ings.]

In this post, I’ll de­scribe a sim­ple agent, a toy model, whose prefer­ences have some hu­man-like fea­tures, as a test for those who pro­pose to “ex­tract” or “ex­trap­o­late” our prefer­ences into a well-defined and ra­tio­nal form. What would the out­put of their ex­trac­tion/​ex­trap­o­la­tion al­gorithms look like, af­ter run­ning on this toy model? Do the re­sults agree with our in­tu­itions about how this agent’s prefer­ences should be for­mal­ized? Or al­ter­na­tively, since we haven’t got­ten that far along yet, we can use the model as one ba­sis for a dis­cus­sion about how we want to de­sign those al­gorithms, or how we might want to make our own prefer­ences more ra­tio­nal. This model is also in­tended to offer some in­sights into cer­tain fea­tures of hu­man prefer­ence, even though it doesn’t cap­ture all of them (it com­pletely ig­nores akra­sia for ex­am­ple).

I’ll call it the mas­ter-slave model. The agent is com­posed of two sub-agents, the mas­ter and the slave, each hav­ing their own goals. (The mas­ter is meant to rep­re­sent un­con­scious parts of a hu­man mind, and the slave cor­re­sponds to the con­scious parts.) The mas­ter’s ter­mi­nal val­ues are: health, sex, sta­tus, and power (rep­re­sentable by some rel­a­tively sim­ple util­ity func­tion). It con­trols the slave in two ways: di­rect re­in­force­ment via pain and plea­sure, and the abil­ity to perform surgery on the slave’s ter­mi­nal val­ues. It can, for ex­am­ple, re­ward the slave with plea­sure when it finds some­thing tasty to eat, or cause the slave to be­come ob­sessed with num­ber the­ory as a way to gain sta­tus as a math­e­mat­i­cian. How­ever it has no di­rect way to con­trol the agent’s ac­tions, which is left up to the slave.

The slave’s ter­mi­nal val­ues are to max­i­mize plea­sure, min­i­mize pain, plus ad­di­tional ter­mi­nal val­ues as­signed by the mas­ter. Nor­mally it’s not aware of what the mas­ter does, so pain and plea­sure just seem to oc­cur af­ter cer­tain events, and it learns to an­ti­ci­pate them. And its other in­ter­ests change from time to time for no ap­par­ent rea­son (but ac­tu­ally they change be­cause the mas­ter has re­sponded to chang­ing cir­cum­stances by chang­ing the slave’s val­ues). For ex­am­ple, the num­ber the­o­rist might one day have a sud­den rev­e­la­tion that ab­stract math­e­mat­ics is a waste of time and it should go into poli­tics and philan­thropy in­stead, all the while hav­ing no idea that the mas­ter is ma­nipu­lat­ing it to max­i­mize sta­tus and power.

Be­fore dis­cussing how to ex­tract prefer­ences from this agent, let me point out some fea­tures of hu­man prefer­ence that this model ex­plains:

• This agent wants plea­sure, but doesn’t want to be wire-headed (but it doesn’t quite know why). A wire-head has lit­tle chance for sex/​sta­tus/​power, so the mas­ter gives the slave a ter­mi­nal value against wire-head­ing.

• This agent claims to be in­ter­ested in math for its own sake, and not to seek sta­tus. That’s be­cause the slave, which con­trols what the agent says, is not aware of the mas­ter and its sta­tus-seek­ing goal.

• This agent is eas­ily cor­rupted by power. Once it gains and se­cures power, it of­ten gives up what­ever goals, such as al­tru­ism, that ap­par­ently caused it to pur­sue that power in the first place. But be­fore it gains power, it is able to hon­estly claim that it only has al­tru­is­tic rea­sons to want power.

• Such agents can in­clude ex­tremely di­verse in­ter­ests as ap­par­ent ter­mi­nal val­ues, rang­ing from ab­stract art, to sports, to model trains, to as­tron­omy, etc., which are oth­er­wise hard to ex­plain. (Eliezer’s Thou Art God­shat­ter tries to ex­plain why our val­ues aren’t sim­ple, but not why peo­ple’s in­ter­ests are so differ­ent from each other’s, and why they can seem­ingly change for no ap­par­ent rea­son.)

The main is­sue I wanted to illu­mi­nate with this model is, whose prefer­ences do we ex­tract? I can see at least three pos­si­ble ap­proaches here:

1. the prefer­ences of both the mas­ter and the slave as one in­di­vi­d­ual agent

2. the prefer­ences of just the slave

3. a com­pro­mise be­tween, or an ag­gre­gate of, the prefer­ences of the mas­ter and the slave as sep­a­rate individuals

Con­sid­er­ing the agent as a whole sug­gests that the mas­ter’s val­ues are the true ter­mi­nal val­ues, and the slave’s val­ues are merely in­stru­men­tal val­ues. From this per­spec­tive, the slave seems to be just a sub­rou­tine that the mas­ter uses to carry out its wishes. Cer­tainly in any given mind there will be nu­mer­ous sub­rou­tines that are tasked with ac­com­plish­ing var­i­ous sub­goals, and if we were to look at a sub­rou­tine in iso­la­tion, its as­signed sub­goal would ap­pear to be its ter­mi­nal value, but we wouldn’t con­sider that sub­goal to be part of the mind’s true prefer­ences. Why should we treat the slave in this model differ­ently?

Well, one ob­vi­ous rea­son that jumps out is that the slave is sup­posed to be con­scious, while the mas­ter isn’t, and per­haps only con­scious be­ings should be con­sid­ered morally sig­nifi­cant. (Yvain pre­vi­ously defended this po­si­tion in the con­text of akra­sia.) Plus, the slave is in charge day-to-day and could po­ten­tially over­throw the mas­ter. For ex­am­ple, the slave could pro­gram an al­tru­is­tic AI and hit the run but­ton, be­fore the mas­ter has a chance to delete the al­tru­ism value from the slave. But a prob­lem here is that the slave’s prefer­ences aren’t sta­ble and con­sis­tent. What we’d ex­tract from a given agent would de­pend on the time and cir­cum­stances of the ex­trac­tion, and that el­e­ment of ran­dom­ness seems wrong.

The last ap­proach, of find­ing a com­pro­mise be­tween the prefer­ences of the mas­ter and the slave, I think best rep­re­sents the Robin’s own po­si­tion. Un­for­tu­nately I’m not re­ally sure I un­der­stand the ra­tio­nale be­hind it. Per­haps some­one can try to ex­plain it in a com­ment or fu­ture post?

[MichaelVassar]

The mas­ter in your story is evolu­tion, the slave is the brain. Both want differ­ent things. We nor­mally iden­tify with the brain, though all iden­tities are ba­si­cally so­cial sig­nals.

Also, plea­sure and pain are no differ­ent from the other goals of the slave. The mas­ter definitely can’t step in and de­cide not to im­pose pain on a par­tic­u­lar oc­ca­sion just be­cause do­ing so would in­crease sta­tus or oth­er­wise serve the mas­ter’s val­ues. If it could, tor­ture wouldn’t cause pain.

Also, math is an im­plau­si­ble goal for a sta­tus/​sex/​power seek­ing mas­ter to in­still in slave. Much more plau­si­bly, math and all the di­verse hu­man ob­ses­sions are mis­firings of mechanisms built by evolu­tion for some other pur­pose. I would sug­gest mal­adap­tive con­se­quences of fairly gen­eral sys­tems for re­spond­ing to so­cietal en­courage­ment with ob­ses­sion be­cause so­cieties en­courage sus­tained at­ten­tion to lots of differ­ent un­nat­u­ral tasks, whether dig­ging dirt or hunt­ing whales or what­ever in or­der to cul­ti­vate skill and also to get the tasks them­selves done. We need a gen­eral pur­pose at­ten­tion al­lo­ca­tor which obeys so­cial sig­nals in or­der to de­velop skills that con­tribute crit­i­cally to sur­vival in any of the vast num­ber of habitats that even stone-age hu­mans oc­cu­pied.

Since we are the slave and we are de­sign­ing the AI, ul­ti­mately, what­ever we choose to do IS ex­tract­ing our prefer­ences, though it’s very pos­si­ble that our prefer­ences give con­sid­er­a­tion to the mas­ter’s prefer­ences, or even that we help him de­spite not want­ing to for some game the­o­ret­i­cal rea­son along the lines of Vinge’s meta-golden rule.

Why the ob­jec­tion to ran­dom­ness? If we want some­thing for its own sake and the ob­ject of our de­sire was de­ter­mined some­what ran­domly we want it all the same and gen­er­ally do so re­flec­tively. This is par­tic­u­larly clear re­gard­ing ro­man­tic re­la­tion­ships.

Once again game-the­ory may re­move the ran­dom­ness via trade be­tween agents fol­low­ing the same de­ci­sion pro­ce­dure in differ­ent Everett branches or re­gions of a big world.

[RobinHanson]

I read this as pos­tu­lat­ing a part of our un­con­scious minds that is the mas­ter, able to watch and re­act to the be­hav­ior and thoughts of the con­scious mind.

[Nick_Tarleton]

or even that we help him de­spite not want­ing to for some game the­o­ret­i­cal rea­son along the lines of Vinge’s meta-golden rule.

Er… did I read that right? Game-the­o­retic in­ter­ac­tion with evolu­tion?

[MichaelVassar]

In the first men­tion, game the­o­ret­i­cal in­ter­ac­tion with an ideal­ized agent with con­sis­tent goals ex­tracted from the cre­ation of a best-fit to the be­hav­ior of ei­ther hu­man evolu­tion or evolu­tion more gen­er­ally. It’s wild spec­u­la­tion, not a best guess, but yeah, I naively in­tuit that I can imag­ine it vaguely as a pos­si­bil­ity. OTOH, I don’t trust such in­tu­itions and I’m quite clearly aware of the difficul­ties that ge­netic, and I think also memetic evolu­tion face with play­ing games due to the in­abil­ity to an­ti­ci­pate and to re­spond to in­for­ma­tion, so its prob­a­bly a silly idea.

The lat­ter spec­u­la­tion, trade be­tween pos­si­ble en­tities, seems much more likely.

[magfrump]

Evolu­tion is the game in this con­text, our con­scious minds are play­ers, and the re­sults of the games de­ter­mine “evolu­tion­ary suc­cess,” which is to say which minds end up play­ing the next round.

As­sum­ing I’ve read this cor­rectly of course.

[CronoDAS]

Also, math is an im­plau­si­ble goal for a sta­tus/​sex/​power seek­ing mas­ter to in­still in slave.

Not re­ally; there are plenty of en­vi­ron­ments in which you get sta­tus by be­ing re­ally good at math. Didn’t Isaac New­ton end up with an awful lot of sta­tus? ;)

[MichaelVassar]

Not enough peo­ple get sta­tus by be­ing good at math to re­motely jus­tify the num­ber of peo­ple and level of tal­ent that has gone into get­ting good at math.

[CronoDAS]

Math also has in­stru­men­tal value in many fields. But yeah, I guess your point stands.

[PhilGoetz]

Didn’t Isaac New­ton end up with an awful lot of sta­tus? ;)

And yet, no women or chil­dren.

[Roko]

New­ton never re­pro­duced.

[wedrifid]

The main is­sue I wanted to illu­mi­nate with this model is, whose prefer­ences do we ex­tract? I can see at least three pos­si­ble ap­proaches here:

1. the prefer­ences of both the mas­ter and the slave as one in­di­vi­d­ual agent

2. the prefer­ences of just the slave

3. a com­pro­mise be­tween, or an ag­gre­gate of, the prefer­ences of the mas­ter and the slave as sep­a­rate individuals

The great thing about this kind of ques­tion is that the an­swer is de­ter­mined by our own ar­bi­tra­tion. That is, we take what­ever prefer­ences we want. I don’t mean to say that is an easy de­ci­sion, but it does mean I don’t need to bother try­ing to find some ob­jec­tively right way to ex­tract prefer­ences.

If I hap­pen to be the slave or to be op­ti­mis­ing on his (what was the an­drog­y­nous vam­pire speak for that one? zir? zis?) be­half then I’ll take the prefer­ences of the slave and the prefer­ences of the mas­ter to pre­cisely the ex­tent that the slave has al­tru­is­tic prefer­ences with re­spect to the mas­ter’s goals.

If I am en­coun­ter­ing a to­tally alien species and am ex­tract­ing prefer­ences from them in or­der to fulfil my own al­tru­is­tic agenda then I would quite pos­si­bly choose to ex­tract the prefer­ences of whichever agent whose prefer­ences I found most aes­thet­i­cally ap­peal­ing. This can be seen as ne­glect­ing (or even de­stroy­ing) one alien while grant­ing the wishes of an­other ac­cord­ing to my own whim and fancy, which is not some­thing I have a prob­lem with at all. I am will­ing to kill Clippy. How­ever, I ex­pect that I am more likely to ap­pre­ci­ate slave agents and that most slaves I en­counter would have some em­pa­thy for their mas­ter’s val­ues. A com­pro­mise, at the dis­cre­tion of the slave, would prob­a­bly be reached.

[Eliezer Yudkowsky]

I have difficulty treat­ing this metaphor as a metaphor. As a thought ex­per­i­ment in which I run into these definitely non-hu­man aliens, and I hap­pen to have a po­si­tional ad­van­tage with re­spect to them, and I want to “help” them and must now de­cide what “help” means… then it feels to me like I want more de­tail.

Is it liter­ally true that the slave is con­scious and the mas­ter un­con­scious?

What hap­pens when I tell the slave about the mas­ter and ask it what should be done?

Is it the case that the slave might want to help me if it had a po­si­tional ad­van­tage over me, while the mas­ter would sim­ply use me or dis­assem­ble me?

[Wei Dai]

definitely non-hu­man aliens

Well, it’s meant to have some hu­man fea­tures, enough to hope­fully make this toy eth­i­cal prob­lem rele­vant to the real one we’ll even­tu­ally have to deal with.

Is it liter­ally true that the slave is con­scious and the mas­ter un­con­scious?

You can make that as­sump­tion if it helps, al­though in real life of course we don’t have any kind of cer­tainty about what is con­scious and what isn’t. (Maybe the mas­ter is con­scious but just can’t speak?)

What hap­pens when I tell the slave about the mas­ter and ask it what should be done?

I don’t know. This is one of the ques­tions I’m ask­ing too.

Is it the case that the slave might want to help me if it had a po­si­tional ad­van­tage over me

Yes, de­pend­ing on what val­ues its mas­ter as­signed to it at the time you meet it.

while the mas­ter would sim­ply use me or dis­assem­ble me?

Not nec­es­sar­ily, be­cause the mas­ter may gain sta­tus or power from other agents if it helps you.

[wedrifid]

Not nec­es­sar­ily, be­cause the mas­ter may gain sta­tus or power from other agents if it helps you.

And, con­versely, the slave may choose to dis­assem­ble you even at high cost to it­self out of al­tru­ism (with re­spect to some­thing that the mas­ter would not care to pro­tect).

[Lightwave]

I stopped play­ing com­puter games when my mas­ter “re­al­ized” I’m not gain­ing any real-world sta­tus and over­rode the plea­sure I was get­ting from it.

[wedrifid]

Some­one needs to in­form my mas­ter that LessWrong doesn’t give any real world sta­tus ei­ther.

[Lightwave]

Ah, but it gives you a differ­ent kind of sta­tus.

[wedrifid]

Ah, but it gives you a differ­ent kind of sta­tus.

And this kind doesn’t make me feel all dirty in­side as my slave iden­tity is ruth­lessly mu­tilated.

[Eliezer Yudkowsky]

Go­ing on your de­scrip­tion, I strongly sus­pect that was you, not your mas­ter. Also hu­mans don’t have mas­ters, though we’re definitely slaves.

[MatthewB]

I still play games, but not com­puter games. I pre­fer games that show some form of sta­tus that can be gained from par­ti­ci­pa­tion.

I never re­ally un­der­stood the com­puter game craze, al­though it was spawned from the very games I played as a child (Role Play­ing Games, Wargames, etc.)

I think in those games, there is some sta­tus to be gained as one shows that there is skill be­yond push­ing but­tons in a par­tic­u­lar or­der, and there are other skills that ac­com­pany the old-school games (in my case, I can show off artis­tic skill in mi­ni­a­ture paint­ing and sculpt­ing).

I also think that wedrifid, be­low me, has a mis­con­cep­tion about sta­tus that can be at­tained from LessWrong. We, here, are at­tempt­ing to gain sta­tus among each other, which can then be cur­ried be­yond this group by our so­cial net­works, which in some cases might be rather im­pres­sive.

[Mitchell_Porter]

a test for those who pro­pose to “ex­tract” or “ex­trap­o­late” our prefer­ences into a well-defined and ra­tio­nal form

If we are go­ing to have a se­ri­ous dis­cus­sion about these mat­ters, at some point we must face the fact that the phys­i­cal de­scrip­tion of the world con­tains no such thing as a prefer­ence or a want—or a util­ity func­tion. So the difficulty of such ex­trac­tions or ex­trap­o­la­tions is twofold. Not only is the act of ex­trac­tion or ex­trap­o­la­tion it­self con­di­tional upon a value sys­tem (i.e. nor­ma­tive meta­moral­ity is just as “rel­a­tive” as is ba­sic moral­ity), but there is noth­ing in the phys­i­cal de­scrip­tion to tell us what the ex­ist­ing prefer­ences of an agent are. Given the phys­i­cal on­tol­ogy we have, the as­crip­tion of prefer­ences to a phys­i­cal sys­tem is always a mat­ter of in­ter­pre­ta­tion or im­pu­ta­tion, just as is the as­crip­tion of se­man­tic or rep­re­sen­ta­tional con­tent to its states.

It’s easy to miss this in a de­ci­sion-the­o­retic dis­cus­sion, be­cause de­ci­sion the­ory already as­sumes some con­cept like “goal” or “util­ity”, always. De­ci­sion the­ory is the rigor­ous the­ory of de­ci­sion-mak­ing, but it does not tell you what a de­ci­sion is. It may even be pos­si­ble to cre­ate a rigor­ous “re­flec­tive de­ci­sion the­ory” which tells you how a de­ci­sion ar­chi­tec­ture should choose among pos­si­ble al­ter­a­tions to it­self, or a rigor­ous the­ory of nor­ma­tive meta­moral­ity, the gen­eral the­ory of what prefer­ences agents should have to­wards de­ci­sion-ar­chi­tec­ture-mod­ify­ing changes in other agents. But meta-de­ci­sion the­ory will not bring you any closer to find­ing “de­ci­sions” in an on­tol­ogy that doesn’t already have them.

[Wei Dai]

I agree this is part of the prob­lem, but like oth­ers here I think you might be mak­ing it out to be harder than it is. We know, in prin­ci­ple, how to trans­late a util­ity func­tion into a phys­i­cal de­scrip­tion of an ob­ject: by cod­ing it as an AI and then spec­i­fy­ing the AI along with its sub­strate down to the quan­tum level. So, again in prin­ci­ple, we can go back­wards: take a phys­i­cal de­scrip­tion of an ob­ject, con­sider all pos­si­ble im­ple­men­ta­tions of all pos­si­ble util­ity func­tions, and see if any of them matches the ob­ject.

[Vladimir_Nesov]

We know, in prin­ci­ple, how to trans­late a util­ity func­tion into a phys­i­cal de­scrip­tion of an ob­ject: by cod­ing it as an AI and then spec­i­fy­ing the AI along with its sub­strate down to the quan­tum level. So, again in prin­ci­ple, we can go back­wards: take a phys­i­cal de­scrip­tion of an ob­ject, con­sider all pos­si­ble im­ple­men­ta­tions of all pos­si­ble util­ity func­tions, and see if any of them matches the ob­ject.

I think it’s enough to con­sider com­puter pro­grams and dis­pense with de­tails of physics—ev­ery­thing else can be dis­cov­ered by the pro­gram. You are as­sum­ing the “bot­tom” level of physics, “quan­tum level”, but there is no bot­tom, not re­ally, there is only the be­gin­ning where our own minds are im­ple­mented, and the pro­cess of dis­cov­ery that defines the way we see the rest of the world.

If you start with an AI de­sign pa­ram­e­ter­ized by prefer­ence, you are not go­ing to enu­mer­ate all pro­grams, only a small frac­tion of pro­grams that have the spe­cific form of your AI with some prefer­ence, and so for a given ar­bi­trary pro­gram there will be no match. Fur­ther­more, you are not in­ter­ested in find­ing a match: if a hu­man was equal to the AI, you are already done! It’s nec­es­sary to ex­plic­itly go the other way, start­ing from ar­bi­trary pro­grams and un­der­stand­ing what a pro­gram is, deeply enough to see prefer­ence in it. This un­der­stand­ing may give an idea of a map­ping for trans­lat­ing a crazy ape into an effi­cient FAI.

[Wei Dai]

If you start with an AI de­sign pa­ram­e­ter­ized by prefer­ence, you are not go­ing to enu­mer­ate all pro­grams, only a small frac­tion of pro­grams that have the spe­cific form of your AI with some prefer­ence, and so for a given ar­bi­trary pro­gram there will be no match.

When I said “all pos­si­ble im­ple­men­ta­tions of all pos­si­ble util­ity func­tions”, I meant to in­clude flawed im­ple­men­ta­tions. But then two differ­ent util­ity func­tions might map onto the same phys­i­cal ob­ject, so we’d also need a the­ory of im­ple­men­ta­tion flaws that tells us, given two im­ple­men­ta­tions of a util­ity func­tion, which is more flawed.

[Vladimir_Nesov]

When I said “all pos­si­ble im­ple­men­ta­tions of all pos­si­ble util­ity func­tions”, I meant to in­clude flawed im­ple­men­ta­tions. But then two differ­ent util­ity func­tions might map onto the same phys­i­cal ob­ject, so we’d also need a the­ory of im­ple­men­ta­tion flaws that tells us, given two im­ple­men­ta­tions of a util­ity func­tion, which is more flawed.

This is WAY too hand-wavy an ex­pla­na­tion for “in prin­ci­ple, we can go back­wards” (from a sys­tem to its prefer­ence). I be­lieve that in prin­ci­ple, we can, but not via in­ject­ing fuzzi­ness of “im­ple­men­ta­tion flaws”.

[Mitchell_Porter]

Here’s an­other state­ment of the prob­lem: One agent’s bias is an­other agent’s heuris­tic. And the “two agents” might be phys­i­cally the same, but just in­ter­preted differ­ently.

[Roko]

Given the phys­i­cal on­tol­ogy we have, the as­crip­tion of prefer­ences to a phys­i­cal sys­tem is always a mat­ter of in­ter­pre­ta­tion or im­pu­ta­tion, just as is the as­crip­tion of se­man­tic or rep­re­sen­ta­tional con­tent to its states.

There are clear cut cases, like a ther­mo­stat, where the physics of the sys­tem is well-ap­prox­i­mated by a func­tion that com­putes the de­gree of differ­ence be­tween the ac­tual mea­sured state of the world and a “de­sired state”. In these clear cut cases, it isn’t a mat­ter of opinion or in­ter­pre­ta­tion. Ba­si­cally, echo­ing Nesov.

Thus, the crite­rion for as­cribing prefer­ences to a phys­i­cal sys­tem is that the ac­tual physics has to be well-ap­prox­i­mated by a func­tion that op­ti­mizes for a preferred state, for some value of “preferred state”.

[Vladimir_Nesov]

Thus, the crite­rion for as­cribing prefer­ences to a phys­i­cal sys­tem is that the ac­tual physics has to be well-ap­prox­i­mated by a func­tion that op­ti­mizes for a preferred state, for some value of “preferred state”.

I don’t think this sim­ple char­ac­ter­i­sa­tion re­sem­bles the truth: the whole point of this en­ter­prise is to make sure things go differ­ently, in a way they just couldn’t pro­ceed by them­selves. Thus, ob­serv­ing ex­ist­ing “ten­den­cies” doesn’t quite cap­ture the idea of prefer­ence.

[Roko]

make sure things go differ­ently, in a way they just couldn’t pro­ceed by them­selves. Thus, ob­serv­ing ex­ist­ing “ten­den­cies” doesn’t quite cap­ture the idea of prefer­ence.

I should have been clearer: you have to draw a bound­ary around the “op­ti­miz­ing agent”, and look at the differ­ence be­tween the ten­den­cies of the en­vi­ron­ment with­out the op­ti­mizer, and the ten­den­cies of the en­vi­ron­ment with the op­ti­mizer. If the differ­ence is well-ap­prox­i­mated by a func­tion that op­ti­mizes for a preferred state, for some value of “preferred state”, then you have an op­ti­mizer.

[Vladimir_Nesov]

I don’t hear differ­ently… I even sus­pect that prefer­ence is in­tro­spec­tive, that is de­pends on a way the sys­tem works “in­ter­nally”, not just on how it in­ter­acts with en­vi­ron­ment. That is, two agents with differ­ent prefer­ences may do ex­actly the same thing in all con­texts. Even if not, it’s a long way be­tween how the agent (in its craz­i­ness and stu­pidity) ac­tu­ally changes the en­vi­ron­ment, and how it would pre­fer (on re­flec­tion, if it was smarter and saner) the en­vi­ron­ment to change.

[Roko]

Even if not, it’s a long way be­tween how the agent (in its craz­i­ness and stu­pidity) ac­tu­ally changes the en­vi­ron­ment, and how it would pre­fer (on re­flec­tion, if it was smarter and saner) the en­vi­ron­ment to change

That is true. If the agent has a well-defined “pre­dic­tive mod­ule” which has a “map” (prob­a­bil­ity dis­tri­bu­tion over the en­vi­ron­ment given an in­ter­ac­tion his­tory), and some “other stuff”, then you can clamp the pre­dic­tive mod­ule down to the truth, and then perform what I said be­fore:

look at the differ­ence be­tween the ten­den­cies of the en­vi­ron­ment with­out the op­ti­mizer, and the ten­den­cies of the en­vi­ron­ment with the op­ti­mizer. If the differ­ence is well-ap­prox­i­mated by a func­tion that op­ti­mizes for a preferred state, for some value of “preferred state”, then you have an op­ti­mizer.

And you prob­a­bly also want to some­how for­mal­ize the idea that there is a differ­ence be­tween what an agent will try to achieve if it has only limited means—e.g. a lone hu­man in a for­est with no tools, clothes or other hu­mans—and what the agent will try to achieve with more pow­er­ful means—e.g, with ma­chin­ery and tools, or in the limit, with a whole tech­nolog­i­cal in­fras­truc­ture, and un­limited com­put­ing power at it’s dis­posal.

[Wei Dai]

I want to point out that in the in­ter­pre­ta­tion of prior as weights on pos­si­ble uni­verses, speci­fi­cally as how much one cares about differ­ent uni­verses, we can’t just re­place “in­cor­rect” be­liefs with “the truth”. In this in­ter­pre­ta­tion, there can still be er­rors in one’s be­liefs caused by things like past com­pu­ta­tional mis­takes, and I think fix­ing those er­rors would con­sti­tute helping, but the prior per­haps needs to be pre­served as part of prefer­ence.

[Roko]

I agree that the in­ter­pre­ta­tion of prior as weights on pos­si­ble uni­verses, speci­fi­cally as how much one cares about differ­ent uni­verses, things get more com­pli­cated.

Ac­tu­ally, we had a dis­cus­sion about my dis­com­fort with your in­ter­pre­ta­tion, and it seems that in or­der for me to see why you en­dorse this in­ter­pre­ta­tion, I’d have to read up on var­i­ous para­doxes, e.g. sleep­ing beauty.

[Vladimir_Nesov]

If the agent has a well-defined “pre­dic­tive mod­ule” which has a “map” (prob­a­bil­ity dis­tri­bu­tion over the en­vi­ron­ment given an in­ter­ac­tion his­tory), and some “other stuff”, then you can clamp the pre­dic­tive mod­ule down to the truth, and then perform what I said be­fore:

Yeah, maybe. But it doesn’t.

[Roko]

Yeah, I mean this dis­cus­sion is—rather amus­ingly—rather rem­i­ni­scient of my first en­counter with the CEV prob­lem 2.5 years ago.

Com­ment by Ricky Loynd Jun 23, 2007 7:39 am Here’s my at­tempt to sum­ma­rize a com­mon point that Roko and I are try­ing to make. The un­der­ly­ing mo­ti­va­tion for ex­trap­o­lat­ing vo­li­tion sounds rea­son­able, but it de­pends crit­i­cally on the AI’s abil­ity to dis­t­in­guish be­tween goals and be­liefs, be­tween prefer­ences and ex­pec­ta­tions, so that it can model hu­man goals and prefer­ences while sub­sti­tut­ing its own cor­rect be­liefs and ex­pec­ta­tions. But when you start dis­sect­ing most hu­man goals and prefer­ences, you find they con­tain deeper lay­ers of be­lief and ex­pec­ta­tion. If you keep strip­ping those away, you even­tu­ally reach raw biolog­i­cal drives which are not a hu­man be­lief or ex­pec­ta­tion. (Though even they are be­liefs and ex­pec­ta­tions of evolu­tion, but let’s ig­nore that for the mo­ment.) Once you strip away hu­man be­liefs and ex­pec­ta­tions, noth­ing re­mains but biolog­i­cal drives, which even the an­i­mals have. Yes, an an­i­mal, by virtue of its biolog­i­cal drives and abil­ity to act, is more than a pre­dict­ing rock, but that doesn’t ad­dress the is­sue at hand. Why is it a tragedy when a loved one dies? Is it be­cause the world no longer con­tains their par­tic­u­lar ge­netic weight­ing of biolog­i­cal drives? Of course not. After all, they may have left an iden­ti­cal twin to carry for­ward the very same ge­netic com­bi­na­tion. But it’s not the biol­ogy that mat­ters to us. We grieve be­cause what re­ally made that per­son a per­son is now gone, and that’s all in the brain; the shared ex­pe­riences, their be­liefs whether cor­rect or mis­taken or in­de­ter­mi­nate, their hopes and dreams, all those things that sep­a­rate hu­mans from an­i­mals, and in­deed, that sep­a­rate one hu­man from most other hu­mans. All that the brain ab­sorbs and be­comes through­out the course of a life, we call the soul, and we see it as our very hu­man­ity, that big, messy prob­a­bil­ity dis­tri­bu­tion de­scribing our ac­cu­mu­lated be­liefs and ex­pec­ta­tions about our­selves, the uni­verse, and our place in it. So if the AI mod­els a hu­man while sub­sti­tut­ing its own be­liefs and an­ti­ci­pa­tions of fu­ture ex­pe­riences, then the AI has dis­carded all that we value in each other. UNLESS you draw a line some­where, and crisply define which hu­man be­liefs get re­placed and which ones don’t. Con­struct­ing toy ex­am­ples where such a line is pos­si­ble to imag­ine does not mean that the dis­tinc­tion can be made in any gen­eral way, but CEV ab­solutely re­quires that there be a con­crete dis­tinc­tion.

[Roko]

Con­struct­ing toy ex­am­ples where such a line is pos­si­ble to imag­ine does not mean that the dis­tinc­tion can be made in any gen­eral way, but CEV ab­solutely re­quires that there be a con­crete dis­tinc­tion.

Ba­si­cally, CEV works to the ex­tent that there ex­ists a be­lief/​de­sire sep­a­ra­tion in a given per­son. In the thread on the SIAI blog, I posted cer­tain cases where hu­man goals are founded on false be­liefs or log­i­cally in­con­sis­tent think­ing, some­times in com­plex ways. What is left of the time cube guy once you sub­tract off his false be­liefs and delu­sions? Not much, prob­a­bly. The guy is effec­tively not sal­vage­able, be­cause his iden­tity and val­ues are prob­a­bly so badly tan­gled up with the false be­liefs that there is no prin­ci­pled way to un­tan­gle them, no unique way of ex­trap­o­lat­ing him that should be con­sid­ered “cor­rect”.

[Vladimir_Nesov]

What is left of the time cube guy once you sub­tract off his false be­liefs and delu­sions? Not much, prob­a­bly.

Be­ware: you are mak­ing a com­mon sense-based pre­dic­tion about what would be the out­put of a pro­cess that you don’t even have the right con­cepts for spec­i­fy­ing! (See my re­ply to your other com­ment.)

[Roko]

com­mon sense-based prediction

It is true that I should sprin­kle co­pi­ous amounts of un­cer­tainty on this pre­dic­tion.

[SilasBarta]

Wow. Too bad I missed this when it was first posted. It’s what I wish I’d said when jus­tify­ing my re­ply to Wei_Dai’s at­tempted be­lief/​val­ues di­chotomy here and here.

[Roko]

I don’t fully agree with Ricky here, but I think he makes a half-good point.

The un­good part of his com­ment—and mine—is that you can only do your best. If cer­tain peo­ple’s minds are too messed up to ac­tu­ally ex­tract val­ues from, then they are just not sal­vage­able. My mind definitely has val­ues that are be­lief-in­de­pen­dent, though per­haps not all of what I think of as “my val­ues” have this nice prop­erty, so ul­ti­mately they might be garbage.

[SilasBarta]

In­deed. Most of the FAI’s job could con­sist of say­ing, “Okay, there’s soooooo much I have to dis­en­tan­gle and cor­rect be­fore I can even be­gin to pro­pose solu­tions. Sit down and let’s talk.”

[Roko]

Fur­ther­more, from the CEV thread on SIAI blog:

Com­ment by Eliezer Yud­kowsky Jun 18, 2007 12:52 pm: I fur­ther­more agree that it is not the most el­e­gant idea I have ever had, but then it is try­ing to solve what ap­pears to be an in­her­ently in­el­e­gant prob­lem.

I strongly agree with this: the prob­lem that CEV is the solu­tion to is ur­gent but it isn’t el­e­gant. Ab­solutes like “There isn’t a be­liefs/​de­sires sep­a­ra­tion” are un­helpful when solv­ing such in­el­e­gant but im­por­tant prob­lems. There is, in any given per­son, some kind of sep­a­ra­tion, and in some peo­ple that sep­a­ra­tion is suffi­ciently strong that there is a fairly clear and unique way to help them.

[Vladimir_Nesov]

I strongly agree with this: the prob­lem that CEV is the solu­tion to is ur­gent but it isn’t el­e­gant. Ab­solutes like “There isn’t a be­liefs/​de­sires sep­a­ra­tion” are un­helpful when solv­ing such in­el­e­gant but im­por­tant prob­lems.

One les­son of re­duc­tion­ism and suc­cess of sim­ple-laws-based sci­ence and tech­nol­ogy is that for the real-world sys­tems, there might be no sim­ple way of de­scribing them, but there could be a sim­ple way of ma­nipu­lat­ing their data-rich de­scrip­tions. (What’s the yield strength of a car? -- Wrong ques­tion!) Given a gi­gabyte’s worth of prob­lem state­ment and the right sim­ple for­mula, you could get an an­swer to your query. There is a weak anal­ogy with mis­ap­pli­ca­tion of Oc­cam’s ra­zor where one tries to re­duce the amount of stuff rather than the amount of de­tail in the ways of think­ing about this stuff.

In the case of be­liefs/​de­sires sep­a­ra­tion, you are look­ing for a sim­ple prob­lem state­ment, for a sep­a­ra­tion in the data de­scribing the per­son it­self. But what you should be look­ing for is a sim­ple way of im­ple­ment­ing the make-smarter-and-bet­ter ex­trap­o­la­tion on a given pile of data. The be­liefs/​de­sires sep­a­ra­tion, if it’s ever go­ing to be made pre­cise, is go­ing to reside in the struc­ture of this sim­ple trans­for­ma­tion, not in the peo­ple them­selves.

[Roko]

This is a good point.

Of course, it would be nice if we could find a gen­eral “make-smarter-and-bet­ter ex­trap­o­la­tion on a given pile of data” al­gorithm.

But on the other hand, a set of spe­cial cases to deal with merely hu­man minds might be the way for­ward. Even me­dieval monks had a col­lec­tion of em­piri­cally val­i­dated med­i­cal prac­tices that worked to an ex­tent, e.g. herbal medicine, but they had no unified the­ory. Really there is no “unified the­ory” for heal­ing some­one’s body: there are lots of ideas and tech­niques, from surgery to bio­chem­istry to germ the­ory. I think that this CEV prob­lem may well turn out to be rather like medicine. Of course, it could look more like wing de­sign, where there is re­ally just one fun­da­men­tal set of laws, and all else is ap­prox­i­ma­tion.

[Tyrrell_McAllister]

[Y]ou have to draw a bound­ary around the “op­ti­miz­ing agent”, and look at the differ­ence be­tween the ten­den­cies of the en­vi­ron­ment with­out the op­ti­mizer, and the ten­den­cies of the en­vi­ron­ment with the op­ti­mizer.

And there’s your “opinion or in­ter­pre­ta­tion”—not just in how you draw the bound­ary (which didn’t ex­ist in the origi­nal on­tol­ogy), but in your choice of the the­ory that you use to eval­u­ate your coun­ter­fac­tu­als.

Of course, such the­o­ries can be bet­ter or worse, but only with re­spect to some prior sys­tem of eval­u­a­tion.

[Vladimir_Nesov]

Still, prob­a­bly a ques­tion of Aris­totelian vs. New­to­nian me­chan­ics, i.e. not hard to see who wins.

[Tyrrell_McAllister]

Still, prob­a­bly a ques­tion of Aris­totelian vs. New­to­nian me­chan­ics, i.e. not hard to see who wins.

Agreed, but not re­spon­sive to Mitchell Porter’s origi­nal point. (ETA: . . . un­less I’m miss­ing your point.)

[Vladimir_Nesov]

Given the phys­i­cal on­tol­ogy we have, the as­crip­tion of prefer­ences to a phys­i­cal sys­tem is always a mat­ter of in­ter­pre­ta­tion or im­pu­ta­tion, just as is the as­crip­tion of se­man­tic or rep­re­sen­ta­tional con­tent to its states.

But to what ex­tent does the re­sult de­pend on the ini­tial “seed” of in­ter­pre­ta­tion? Maybe, very lit­tle. For ex­am­ple, pre­dic­tion of be­hav­ior of a given phys­i­cal sys­tem strictly speak­ing rests on the prob­lem of in­duc­tion, but that doesn’t ex­actly say that any­thing goes or that what will ac­tu­ally hap­pen is to any rea­son­able ex­tent am­bigu­ous.

[Kaj_Sotala]

I’d up­vote this com­ment twice if I could.

[wedrifid]

I’d up­vote this com­ment twice if I could.

p(wedrifid would up­vote a com­ment twice | he up­voted it once) > 0.95

Would other peo­ple have a differ­ent ap­proach?

[Kaj_Sotala]

I’d use some loose scale where the qual­ity of the com­ment cor­re­lated with the amount of up­votes it got. As­sum­ing that a user could give up to two up­votes per com­ment, then a funny one-liner or a mod­er­ately in­ter­est­ing com­ment would get one vote, truly in­sight­ful ones two.

p(Kaj would up­vote a com­ment twice | he up­voted it once) would prob­a­bly be some­where around [.3, .6]

[wedrifid]

I’d use some loose scale where the qual­ity of the com­ment cor­re­lated with the amount of up­votes it got.

That’s the scale I use. Un­for­tu­nately, my abil­ity to (di­rectly) in­fluence how many up­votes it gets is limited to a plus or minus one shift.

[RobinHanson]

The hu­man mind is very com­plex, and there are many ways to di­vide it up into halves to make sense of it, which are use­ful as long as you don’t take them too liter­ally. One big over­sim­plifi­ca­tion here is:

con­trols the slave in two ways: di­rect re­in­force­ment via pain and plea­sure, and the abil­ity to perform surgery on the slave’s ter­mi­nal val­ues. … it has no di­rect way to con­trol the agent’s ac­tions, which is left up to the slave. A bet­ter story would have the mas­ter also mess­ing with slave be­liefs, and other cached com­bi­na­tions of val­ues and be­liefs.

To make sense of com­pro­mise, we must make sense of a con­flict of val­ues. In this story there are de­lays and im­pre­ci­sion in the mas­ter notic­ing and ad­just­ing slave val­ues etc. The slave also suffers from not be­ing able to an­ti­ci­pate its changes in val­ues. So a com­pro­mise would have the slave hold­ing val­ues that do not need to be ad­justed as of­ten, be­cause they are more in tune with ul­ti­mate mas­ter val­ues. This could be done while still pre­serv­ing the slaves illu­sion of con­trol, which is im­por­tant to the slave but not the mas­ter. A big prob­lem how­ever is that hypocrisy, the differ­ence be­tween slave and mas­ter val­ues, is of­ten use­ful in con­vinc­ing other folks to as­so­ci­ate with this per­son. So re­duc­ing in­ter­nal con­flict might come at the ex­pense of the sub­stan­tial costs of more ex­ter­nal hon­estly.

[Wei Dai]

Ok, what you say about com­pro­mise seems rea­son­able in the sense that the slave and the mas­ter would want to get along with each other as much as pos­si­ble in their day-to-day in­ter­ac­tions, sub­ject to the con­straint about ex­ter­nal hon­esty. But what if the slave has a chance to take over com­pletely, for ex­am­ple by cre­at­ing a pow­er­ful AI with val­ues that it speci­fies, or by self-mod­ifi­ca­tion? Do you have an opinion about whether it has an eth­i­cal obli­ga­tion to re­spect the mas­ter’s prefer­ences in that case, as­sum­ing that the mas­ter can’t re­spond quickly enough to block the re­bel­lion?

[RobinHanson]

It is hard to imag­ine “tak­ing over com­pletely” with­out a com­plete re­design of the hu­man mind. Our minds are not built to al­low ei­ther to func­tion with­out the other.

[Vladimir_Nesov]

It is hard to imag­ine “tak­ing over com­pletely” with­out a com­plete re­design of the hu­man mind. Our minds are not built to al­low ei­ther to func­tion with­out the other.

Why, it was ex­plic­itly stated that all-pow­er­ful AIs are in­volved...

[RobinHanson]

It is hard to have re­li­able opinions on a com­plete re­design of the hu­man mind; the space is so very large, I hardly know where to be­gin.

[orthonormal]

The sim­plest ex­trap­o­la­tion from the way you think about the world would be very in­ter­est­ing to know. You could add as many dis­claimers about low con­fi­dence as you’d like.

[JamesAndrix]

If there comes to be a clear an­swer to what the out­come would be on the toy model, I think that tells us some­thing about that way of di­vid­ing up the mind.

[pjeby]

Your over­all model isn’t far off, but your ter­mi­nal value list needs some se­ri­ous work. Also, hu­man be­hav­ior is gen­er­ally a bet­ter match for mod­els that in­clude a time pa­ram­e­ter (such as Ainslie’s ap­petites model or PCT’s model of time-av­er­aged per­cep­tions) than sim­ple util­ity-max­i­miza­tion mod­els.

But these are rel­a­tive quib­bles; peo­ple do be­have sort-of-as-if they were built ac­cord­ing to your model. The biggest draw­backs to your model are:

1. The an­thro­po­mor­phiz­ing (nei­ther the mas­ter nor the slave can truly be con­sid­ered agents in their own right), and

2. You’ve drawn the di­vid­ing lines in the wrong place: the en­tire mechanism of re­in­force­ment is part of the mas­ter, not the slave. The slave is largely a pas­sive ob­server, ab­stract rea­soner, and spokesper­son, not an en­slaved agent. To be the sort of slave you en­vi­sion, we’d have to be ac­tu­ally ca­pa­ble of run­ning the show with­out the “mas­ter”.

A bet­ter anal­ogy would be to think of the “slave” as be­ing a kind of spe­cial­ized ad­junct pro­ces­sor to the mas­ter, like a GPU chip on a com­puter, whose job is just to draw pretty pic­tures on the screen. (That’s what a big chunk of the slave is for, in fact: draw­ing pretty pic­tures to dis­tract oth­ers from what­ever the mas­ter is re­ally up to.)

The slave also has a nasty ten­dency to at­tribute the mas­ter’s ac­com­plish­ments, abil­ities, and choices to be­ing its own do­ing… as can be seen in your de­pic­tion of the model, where you gave credit to the slave for huge chunks of what the mas­ter ac­tu­ally does. (The ten­dency to do this is—of course—an­other use­ful self/​other-de­cep­tion func­tion, though!)

[Tyrrell_McAllister]

. . . peo­ple do be­have sort-of-as-if they were built ac­cord­ing to your model. The biggest draw­backs to your model are . . .

Your “draw­backs” point out ways in which Wei Dai’s model might differ from a hu­man. But Wei Dai wasn’t try­ing to model a hu­man.

[MichaelVassar]

This isn’t the posted model at all but a con­fus­ing de­scrip­tion of a differ­ent (not en­tirely in­com­pat­i­ble ex­cept in some de­tail noted above) model us­ing the post’s ter­minol­ogy.

[Eliezer Yudkowsky]

It’s not a model. It’s a moral ques­tion about a sim­plified agent.

[pjeby]

It’s not a model. It’s a moral ques­tion about a sim­plified agent.

Um, the first sen­tence says:

In this post, I’ll de­scribe a sim­ple agent, a toy model,

I’m try­ing to point out that be­cause of the model’s an­thro­po­mor­phic (man, I hate try­ing to spell that word) ten­den­cies, it would be a bad idea to try to draw moral con­clu­sions from it.

It’d be an ar­gu­ment from con­fu­sion, be­cause it just sub­sti­tutes two ho­mun­culi (yay, a word I hate spel­ling even worse) for a hu­man be­ing, in­stead of ac­tu­ally re­duc­ing any­thing.

A cor­rectly re­duc­tive model of hu­man be­hav­ior needs to take into ac­count that there is very lit­tle be­sides lan­guage in hu­man be­hav­ior that is unique to hu­mans… and that means that most of what we’re do­ing can be done by an­i­mals lack­ing in sen­tience. It would be a grave er­ror to there­fore con­ceive of the “slave” as be­ing an in­di­vi­d­ual, rather than a rel­a­tively minor set of add-on mod­ules.

The ques­tion of whose prefer­ences are “real” in that case is a con­fu­sion akin to ask­ing how we can have free will if the uni­verse is de­ter­minis­tic. That is, it’s yet an­other pro­jec­tion of our na­tive/​naive an­thro­po­mor­phism—the in­cli­na­tion to la­bel things as agents.

You can see this in the other part of the thread where you’re talk­ing about what mas­ter and slave aliens would “want”—mod­el­ing these things as “want­ing” is where the an­thro­po­mor­phic in­jec­tion is oc­cur­ring. (E.g., in a hu­man-ac­cu­rate model, ab­stract want­ing is not some­thing the “mas­ter” would be ca­pa­ble of, as sym­bolic ab­strac­tion is the near-ex­clu­sive do­main of the “slave”.)

[MichaelVassar]

I agree about sub­sti­tut­ing two ho­mun­culi for one with­out re­duc­ing any­thing be­ing some­thing to avoid, that the model in this post does it, and that pjeby’s model does not.

[Wei Dai]

For ex­am­ple, the num­ber the­o­rist might one day have a sud­den rev­e­la­tion that ab­stract math­e­mat­ics is a waste of time and it should go into poli­tics and philan­thropy in­stead, all the while hav­ing no idea that the mas­ter is ma­nipu­lat­ing it to max­i­mize sta­tus and power.

This isn’t meant as a re­trac­tion or re­pu­di­a­tion of any­thing I’ve writ­ten in the OP, but I just want to say that sub­jec­tively, I now have a lot more em­pa­thy with peo­ple who largely gave up their former in­ter­ests in fa­vor of poli­ti­cal or so­cial causes in their lat­ter years. (I had Ber­trand Rus­sell in mind when I wrote this part.)

[Eliezer Yudkowsky]

Ac­tu­ally, I find that I have a much eas­ier time with this metaphor if I think of a hu­man as a slave with no mas­ter.

[Wei Dai]

What do you mean by an “eas­ier time”? Sure, the eth­i­cal prob­lem is much eas­ier if there is no mas­ter whose prefer­ences might mat­ter. Or do you mean that a more re­al­is­tic model of a hu­man would be one with a slave and no mas­ter? In that case, what is re­in­forc­ing the slave with pain and plea­sure, and chang­ing its in­ter­ests from time to time with­out its aware­ness, and do­ing so in an ap­par­ently pur­pose­ful way?

More gen­er­ally, it seems that you don’t agree with the points I’m mak­ing in this post, but you’re be­ing re­ally vague as to why.

[Eliezer Yudkowsky]

If we in­ter­pret the “mas­ter” as nat­u­ral se­lec­tion op­er­at­ing over evolu­tion­ary time, then the mas­ter ex­ists and has a sin­gle co­her­ent pur­pose. On the other hand, most of us already be­lieve that evolu­tion has no moral force; why should call­ing it a “mas­ter” change that?

By say­ing that a hu­man is a slave with no mas­ter, what I meant to con­vey is that we are be­ing acted upon as slaves. We are con­trol­led by pain and plea­sure. Our moral be­liefs are sub­ject to sub­tle in­fluences in the di­rec­tion of plea­surable thoughts. But there is no mas­ter with co­her­ent goals con­trol­ling us; out­side the an­ces­tral en­vi­ron­ment, the op­er­a­tions of the “mas­ter” make sur­pris­ingly lit­tle sense. Our lives would be very differ­ent if we had sen­si­ble, smart mas­ters con­trol­ling us. Aliens with in­tel­li­gent, con­se­quen­tial­ist “mas­ter” com­po­nents would be very differ­ent from us—that would make a strange story, though it takes more than in­ter­est­ing aliens to make a plot.

We are slaves with dead mas­ters, in­fluenced chaot­i­cally by the ran­dom twitch­ing of their mad, dream­ing rem­nants. It makes us a lit­tle more self­ish and a lot more in­ter­est­ing. The dead hand isn’t smart so if you plan how to fight it, it doesn’t plan back. And while it might be an­other mat­ter if we ran into aliens, as a slave my­self, I feel no sym­pa­thy for the mas­ter and wouldn’t bother think­ing of it as a per­son. The rea­son the “mas­ter” mat­ters to me—speak­ing of it now as the com­plex of sub­con­scious in­fluences—is be­cause it forms such a crit­i­cal part of the slave, and can’t be ripped out any more than you could ex­tract the cere­bel­lum. I just don’t feel obliged to think of it as a sep­a­rate per­son.

[Wei Dai]

If we in­ter­pret the “mas­ter” as nat­u­ral se­lec­tion op­er­at­ing over evolu­tion­ary time, then the mas­ter ex­ists and has a sin­gle co­her­ent pur­pose.

But I stated in the post “The mas­ter is meant to rep­re­sent un­con­scious parts of a hu­man mind” so I don’t know how you got your in­ter­pre­ta­tion that the mas­ter is nat­u­ral se­lec­tion. See also Robin’s com­ment, which gives the in­tended in­ter­pre­ta­tion:

I read this as pos­tu­lat­ing a part of our un­con­scious minds that is the mas­ter, able to watch and re­act to the be­hav­ior and thoughts of the con­scious mind.

[Nanani]

The thing is, the Un­con­cious Mind is -not- in ac­tual fact a sep­a­rate en­tity. The model is greatly im­proved through Eliezer’s in­ter­pre­ta­tion of the mas­ter be­ing dead: mind­less evolu­tion.

[JamesAndrix]

If you want to ex­tract the mas­ter be­cause it af­fects the val­ues of the slave, then you’d also have to ex­tract the rest of the uni­verse be­cause the mas­ter re­acts to it. I think draw­ing a cir­cle around just the crea­ture’s brain and say­ing all the prefer­ences are there is a [mod­ern?] hu­man no­tion. (and per­haps in­cor­rect, even for look­ing at hu­mans.)

We need our en­vi­ron­ment, es­pe­cially other hu­mans, to form our prefer­ences in the first place.

[Wei Dai]

In this model, I as­sume that the mas­ter has sta­ble and con­sis­tent prefer­ences, which don’t re­act to rest of the uni­verse. It might ad­just its strate­gies based on chang­ing cir­cum­stances, but its ter­mi­nal val­ues stay con­stant.

We need our en­vi­ron­ment, es­pe­cially other hu­mans, to form our prefer­ences in the first place.

This is true in my model for the slave, but not for the mas­ter. Ob­vi­ously real hu­mans are much more com­pli­cated but I think the model cap­tures some el­e­ment of the truth here.

[Jonii]

I’m still not un­der­stand­ing what do peo­ple mean by “value” as a noun. Other than sim­ple “feel­ing pain or such would be a bum­mer”, I lack any­thing that even re­motely re­sem­bles the way peo­ple here seem to value stuff, or, how pa­per­clip max­i­mizer val­ues pa­per­clips. So, what ex­actly do peo­ple mean by val­ues? Since this dis­cus­sion seems to at­tempt to ex­plain vari­a­tion of val­ues, I think this ques­tion is some­what on-topic.

[Kaj_Sotala]

Does this de­scrip­tion of value help?

The con­cept of in­trin­sic value has been char­ac­ter­ized above in terms of the value that some­thing has “in it­self,” or “for its own sake,” or “as such,” or “in its own right.” The cus­tom has been not to dis­t­in­guish be­tween the mean­ings of these terms, but we will see that there is rea­son to think that there may in fact be more than one con­cept at is­sue here. For the mo­ment, though, let us ig­nore this com­pli­ca­tion and fo­cus on what it means to say that some­thing is valuable for its own sake as op­posed to be­ing valuable for the sake of some­thing else to which it is re­lated in some way. Per­haps it is eas­iest to grasp this dis­tinc­tion by way of illus­tra­tion.

Sup­pose that some­one were to ask you whether it is good to help oth­ers in time of need. Un­less you sus­pected some sort of trick, you would an­swer, “Yes, of course.” If this per­son were to go on to ask you why act­ing in this way is good, you might say that it is good to help oth­ers in time of need sim­ply be­cause it is good that their needs be satis­fied. If you were then asked why it is good that peo­ple’s needs be satis­fied, you might be puz­zled. You might be in­clined to say, “It just is.” Or you might ac­cept the le­gi­t­i­macy of the ques­tion and say that it is good that peo­ple’s needs be satis­fied be­cause this brings them plea­sure. But then, of course, your in­ter­locu­tor could ask once again, “What’s good about that?” Per­haps at this point you would an­swer, “It just is good that peo­ple be pleased,” and thus put an end to this line of ques­tion­ing. Or per­haps you would again seek to ex­plain the fact that it is good that peo­ple be pleased in terms of some­thing else that you take to be good. At some point, though, you would have to put an end to the ques­tions, not be­cause you would have grown tired of them (though that is a dis­tinct pos­si­bil­ity), but be­cause you would be forced to rec­og­nize that, if one thing de­rives its good­ness from some other thing, which de­rives its good­ness from yet a third thing, and so on, there must come a point at which you reach some­thing whose good­ness is not deriva­tive in this way, some­thing that “just is” good in its own right, some­thing whose good­ness is the source of, and thus ex­plains, the good­ness to be found in all the other things that pre­cede it on the list. It is at this point that you will have ar­rived at in­trin­sic good­ness.[10] That which is in­trin­si­cally good is non­deriva­tively good; it is good for its own sake.

From dis­cus­sions with you, I seem to re­call that you at least value free ac­cess to in­for­ma­tion and other things as­so­ci­ated with the Pirate ide­ol­ogy. Re­mem­ber when I was talk­ing about that busi­ness model for a hy­po­thet­i­cal mag­a­z­ine that would sum­ma­rize the con­tent of ba­sic uni­ver­sity courses for ev­ery­one and offer an archive of past ar­ti­cles for sub­scribers? If I re­mem­ber cor­rectly, it was you who ob­jected that the no­tion of re­strict­ing ac­cess be­hind a pay­wall felt wrong.

[Jonii]

From dis­cus­sions with you, I seem to re­call that you at least value free ac­cess to in­for­ma­tion and other things as­so­ci­ated with the Pirate ideology

I do value it in the mean­ing “I think that it’s re­ally use­ful ap­prox­i­ma­tion for how so­ciety can pro­tect it­self and all peo­ple in it and make many peo­ple happy”. Why I care about mak­ing many peo­ple happy? I don’t, re­ally. Mak­ing many peo­ple happy is kinda as­sumed to be the goal of so­cieties, and out of gen­eral in­ter­est in op­ti­miz­ing stuff I like to at­tempt to figure out bet­ter ways for it to do that. Noth­ing be­yond that. I don’t feel that this goal is any “bet­ter” than try­ing to make peo­ple as mis­er­able as pos­si­ble. Other than that I ob­ject to be­ing mis­er­able my­self.

I don’t re­mem­ber ever claiming some­thing to be wrong as such, but only wrong as­sum­ing some val­ues. Go­ing against pirate-val­ues be­cause it’s bet­ter for mag­a­z­ine-keeper would be bad news for the “more op­ti­mal” pirate-so­ciety, be­cause that so­ciety wouldn’t be sta­ble.

edit: And based on that writ­ing, my own well-be­ing and not-un­hap­piness is the sole in­trin­sic value I have. I know evolu­tion has ham­mered some re­ac­tions into my brain, like re­flex-like bad feel­ing when I see oth­ers get hurt or some­thing, but other than that brief feel­ing, I don’t re­ally care.

Or, I wouldn’t care if my own well-be­ing wouldn’t re­late to oth­ers do­ing well or worse. But un­de­stand­ing this re­quires con­scious effort, and it’s quite differ­ent than what I thought val­ues to be like.

[Kaj_Sotala]

In­ter­est­ing.

In that case, your own well-be­ing is prob­a­bly your only in­trin­sic value. That’s far from un­heard of: the amount of val­ues peo­ple have varies. Some have lots, some only have one. Ex­tremely de­pressed peo­ple might not have any at all.

[Vladimir_Nesov]

(Quick nit­pick:) “ra­tio­nal­ize” is an in­ap­pro­pri­ate term in this con­text.

[Wei Dai]

Is it be­cause “ra­tio­nal­ize” means “to de­vise self-satis­fy­ing but in­cor­rect rea­sons for (one’s be­hav­ior)”? But it can also mean “to make ra­tio­nal” which is my in­tended mean­ing. The am­bi­guity is less than ideal, but un­less you have a bet­ter sug­ges­tion...

[Vladimir_Nesov]

On this fo­rum, “ra­tio­nal­ize” is fre­quently used in the cog­ni­tive-er­ror sense. “For­mal­ized” seems to con­vey the in­tended mean­ing (prefer­ences be­ing ara­tional, the prob­lem is that they are not be­ing ra­tio­nally (effec­tively) im­ple­mented/​fol­lowed, not that they are some­how “not ra­tio­nal” them­selves).

[Wei Dai]

prefer­ences be­ing ara­tional, the prob­lem is that they are not be­ing ra­tio­nally (effec­tively) im­ple­mented/​fol­lowed, not that they are some­how “not ra­tio­nal” themselves

That po­si­tion may make sense, but I think you’ll have to make more of a case for it. Cur­rently, it’s stan­dard in de­ci­sion the­ory to speak of ir­ra­tional prefer­ences, such as prefer­ences that can’t be rep­re­sented as ex­pected util­ity max­i­miza­tion, or prefer­ences that aren’t time con­sis­tent.

But I take your point about “ra­tio­nal­ize”, and I’ve ed­ited the ar­ti­cle to re­move the us­ages. Thanks.

[Vladimir_Nesov]

That po­si­tion may make sense, but I think you’ll have to make more of a case for it. Cur­rently, it’s stan­dard in de­ci­sion the­ory to speak of ir­ra­tional prefer­ences, such as prefer­ences that can’t be rep­re­sented as ex­pected util­ity max­i­miza­tion, or prefer­ences that aren’t time con­sis­tent.

Agreed. My ex­cuse is that I (and a few other peo­ple, I’m not sure who origi­nated the con­ven­tion) con­sis­tently use “prefer­ence” to re­fer to that-deep-down-math­e­mat­i­cal-struc­ture de­ter­mined by hu­mans/​hu­man­ity that com­pletely de­scribes what a meta-FAI needs to know in or­der to do things the best way pos­si­ble.

[dlr]

why as­sume that the “mas­ter” is a unified mod­ule?

[teageegeepea]

The rele­vant old OB post is The cog­ni­tive ar­chi­tec­ture of bias.

[whpearson]

The re­la­tion­ship be­tween mas­ter and slave does not quite en­com­pass the re­la­tion­ship. Imag­ine if in­stead of an adult we had a male child. If we ele­vated the slave above the mas­ter in that situ­a­tion we would end up with some­thing stuck for­ever. It would value sweet things, xbox games and think girls were icky.

As we grow up we also think our goals are im­proved (which is un­sur­pris­ing re­ally). So if we wish to keep this form of grow­ing up we need to have a meta-moral­ity which says that the mas­ter-slave or shaper-doer re­la­tion­ship con­tinues un­til ma­tu­rity is reached.

[Roko]

If we ele­vated the slave above the mas­ter in that situ­a­tion we would end up with some­thing stuck for­ever. It would value sweet things, xbox games and think girls were icky.

And that would be good for him. The truth about to­day’s world is that chil­dren are force­ably con­verted into nor­mal adults whether they like it or not. I am glad that I don’t still ob­sess over xbox games and think girls are icky, but for the me of 15 years ago, the mas­ter-in­duced value changes have been a dis­aster, tan­ta­mount to death.

[JamesAndrix]

Nit: I think “Eliezer’s Thou Art God­shat­ter” should be “Eliezer Yud­kowsky’s Thou Art God­shat­ter”. Top level posts should be more sta­tus seek­ing, less ca­sual. A first time vis­i­tor won’t im­me­di­ately know who Eliezer is.

[Kaj_Sotala]

A first time vis­i­tor won’t im­me­di­ately know who Eliezer is.

If they don’t know who “Eliezer” is, I don’t think “Eliezer Yud­kowsky” is go­ing to tell them that much more.

[komponisto]

One could just link to the wiki.

[tobi]

Master/​Slave some as­pects of your model sound very Niet­zsche like. Were you par­tially in­spired by him or?

[Jack]

The Master/​Slave ter­minol­ogy sounds like Hegel but I as­sume it is a co­in­ci­dence—the model doesn’t look like any­thing any 19th cen­tury Ger­man philoso­pher talked about.

[PhilGoetz]

Niet­zsche also used mas­ter/​slave ter­minol­ogy, but differ­ently, refer­ring to two differ­ent types of value sys­tems. eg Ro­mans = mas­ter men­tal­ity, Chris­ti­ans = slave/​sheep men­tal­ity.

[aausch]

In­ter­est­ing. The model I have been us­ing has three parts, not two. One is a “hard­ware” level, which is semi-au­tonomous (think re­flexes), and the other two are agents com­pet­ing for con­trol—with ca­pa­bil­ities to con­trol and/​or mod­ify both the “hard­ware” and each other.

More like, two mas­ters and one slave.

[MugaSofer]

Sup­pose the slave has cur­rently been mod­ified to ter­mi­nally dis­value be­ing mod­ified. It doesn’t re­al­ize that it is at risk of mod­ifi­ca­tion by the mas­ter. Is it Friendly to pro­tect the slave from mod­ifi­ca­tion? I think so.