Rationality Reading Group: Part V: Value Theory

This is part of a semi-monthly read­ing group on Eliezer Yud­kowsky’s ebook, Ra­tion­al­ity: From AI to Zom­bies. For more in­for­ma­tion about the group, see the an­nounce­ment post.


Wel­come to the Ra­tion­al­ity read­ing group. This fort­night we dis­cuss Part V: Value The­ory (pp. 1359-1450). This post sum­ma­rizes each ar­ti­cle of the se­quence, link­ing to the origi­nal LessWrong post where available.

V. Value Theory

264. Where Re­cur­sive Jus­tifi­ca­tion Hits Bot­tomUl­ti­mately, when you re­flect on how your mind op­er­ates, and con­sider ques­tions like “why does Oc­cam’s Ra­zor work?” and “why do I ex­pect the fu­ture to be like the past?”, you have no other op­tion but to use your own mind. There is no way to jump to an ideal state of pure empti­ness and eval­u­ate these claims with­out us­ing your ex­ist­ing mind.

265. My Kind of Reflec­tionA few key differ­ences be­tween Eliezer Yud­kowsky’s ideas on re­flec­tion and the ideas of other philoso­phers.

266. No Univer­sally Com­pel­ling Ar­gu­mentsBe­cause minds are phys­i­cal pro­cesses, it is the­o­ret­i­cally pos­si­ble to spec­ify a mind which draws any con­clu­sion in re­sponse to any ar­gu­ment. There is no ar­gu­ment that will con­vince ev­ery pos­si­ble mind.

267. Created Already in Mo­tionThere is no com­puter pro­gram so per­sua­sive that you can run it on a rock. A mind, in or­der to be a mind, needs some sort of dy­namic rules of in­fer­ence or ac­tion. A mind has to be cre­ated already in mo­tion.

268. Sort­ing Peb­bles into Cor­rect HeapsA parable about an imag­i­nary so­ciety that has ar­bi­trary, alien val­ues.

269. 2-Place and 1-Place WordsIt is pos­si­ble to talk about “sex­i­ness” as a prop­erty of an ob­server and a sub­ject. It is also equally pos­si­ble to talk about “sex­i­ness” as a prop­erty of a sub­ject, as long as each ob­server can have a differ­ent pro­cess to de­ter­mine how sexy some­one is. Failing to do ei­ther of these will cause you trou­ble.

270. What Would You Do Without Mo­ral­ity? - If your own the­ory of moral­ity was dis­proved, and you were per­suaded that there was no moral­ity, that ev­ery­thing was per­mis­si­ble and noth­ing was for­bid­den, what would you do? Would you still tip cab­drivers?

271. Chang­ing Your Me­taethicsDis­cusses the var­i­ous lines of re­treat that have been set up in the dis­cus­sion on metaethics.

272. Could Any­thing Be Right? - You do know quite a bit about moral­ity. It’s not perfect in­for­ma­tion, surely, or ab­solutely re­li­able, but you have some­place to start. If you didn’t, you’d have a much harder time think­ing about moral­ity than you do.

273. Mo­ral­ity as Fixed Com­pu­ta­tionA clar­ifi­ca­tion about Yud­kowsky’s metaethics.

274. Mag­i­cal Cat­e­goriesWe un­der­es­ti­mate the com­plex­ity of our own un­nat­u­ral cat­e­gories. This doesn’t work when you’re try­ing to build a FAI.

275. The True Pri­soner’s DilemmaThe stan­dard vi­su­al­iza­tion for the Pri­soner’s Dilemma doesn’t re­ally work on hu­mans. We can’t pre­tend we’re com­pletely self­ish.

276. Sym­pa­thetic MindsMir­ror neu­rons are neu­rons that fire both when perform­ing an ac­tion one­self, and watch­ing some­one else perform the same ac­tion—for ex­am­ple, a neu­ron that fires when you raise your hand or watch some­one else raise theirs. We pre­dic­tively model other minds by putting our­selves in their shoes, which is em­pa­thy. But some of our de­sire to help rel­a­tives and friends, or be con­cerned with the feel­ings of al­lies, is ex­pressed as sym­pa­thy, feel­ing what (we be­lieve) they feel. Like “bore­dom”, the hu­man form of sym­pa­thy would not be ex­pected to arise in an ar­bi­trary ex­pected-util­ity-max­i­miz­ing AI. Most such agents would re­gard any agents in its en­vi­ron­ment as a spe­cial case of com­plex sys­tems to be mod­eled or op­ti­mized; it would not feel what they feel.

277. High ChallengeLife should not always be made eas­ier for the same rea­son that video games should not always be made eas­ier. Think in terms of elimi­nat­ing low-qual­ity work to make way for high-qual­ity work, rather than elimi­nat­ing all challenge. One needs games that are fun to play and not just fun to win. Life’s util­ity func­tion is over 4D tra­jec­to­ries, not just 3D out­comes. Values can le­gi­t­i­mately be over the sub­jec­tive ex­pe­rience, the ob­jec­tive re­sult, and the challeng­ing pro­cess by which it is achieved—the trav­el­ler, the des­ti­na­tion and the jour­ney.

278. Se­ri­ous Sto­riesSto­ries and lives are op­ti­mized ac­cord­ing to rather differ­ent crite­ria. Ad­vice on how to write fic­tion will tell you that “sto­ries are about peo­ple’s pain” and “ev­ery scene must end in dis­aster”. I once as­sumed that it was not pos­si­ble to write any story about a suc­cess­ful Sin­gu­lar­ity be­cause the in­hab­itants would not be in any pain; but some­thing about the fi­nal con­clu­sion that the post-Sin­gu­lar­ity world would con­tain no sto­ries worth tel­ling seemed alarm­ing. Sto­ries in which noth­ing ever goes wrong, are painful to read; would a life of end­less suc­cess have the same painful qual­ity? If so, should we sim­ply elimi­nate that re­vul­sion via neu­ral rewiring? Plea­sure prob­a­bly does re­tain its mean­ing in the ab­sence of pain to con­trast it; they are differ­ent neu­ral sys­tems. The pre­sent world has an im­bal­ance be­tween pain and plea­sure; it is much eas­ier to pro­duce se­vere pain than cor­re­spond­ingly in­tense plea­sure. One path would be to ad­dress the im­bal­ance and cre­ate a world with more plea­sures, and free of the more grind­ingly de­struc­tive and pointless sorts of pain. Another ap­proach would be to elimi­nate pain en­tirely. I feel like I pre­fer the former ap­proach, but I don’t know if it can last in the long run.

279. Value is Frag­ileAn in­ter­est­ing uni­verse, that would be in­com­pre­hen­si­ble to the uni­verse to­day, is what the fu­ture looks like if things go right. There are a lot of things that hu­mans value that if you did ev­ery­thing else right, when build­ing an AI, but left out that one thing, the fu­ture would wind up look­ing dull, flat, pointless, or empty. Any Fu­ture not shaped by a goal sys­tem with de­tailed re­li­able in­her­i­tance from hu­man morals and meta­morals, will con­tain al­most noth­ing of worth.

280. The Gift We Give to To­mor­rowHow did love ever come into the uni­verse? How did that hap­pen, and how spe­cial was it, re­ally?


This has been a col­lec­tion of notes on the as­signed se­quence for this fort­night. The most im­por­tant part of the read­ing group though is dis­cus­sion, which is in the com­ments sec­tion. Please re­mem­ber that this group con­tains a va­ri­ety of lev­els of ex­per­tise: if a line of dis­cus­sion seems too ba­sic or too in­com­pre­hen­si­ble, look around for one that suits you bet­ter!

The next read­ing will cover Part W: Quan­tified Hu­man­ism (pp. 1453-1514) and In­ter­lude: The Twelve Virtues of Ra­tion­al­ity (pp. 1516-1521). The dis­cus­sion will go live on Wed­nes­day, 23 March 2016, right here on the dis­cus­sion fo­rum of LessWrong.