Ghatanathoah

The Mere Cable Chan­nel Ad­di­tion Paradox

Alan Carter on the Com­plex­ity of Value

• Ghatanathoah 2 May 2012 16:42 UTC

  38 points

  on: Ra­tion­al­ity Quotes May 2012

  “It is indeed true that he [Hume] claims that ‘reason is, and ought only to be the slave of the passions.’ But a slave, it should not be forgotten, does virtually all the work.”

  -Alan Carter, Pluralism and Projectivism

• Ghatanathoah 25 Oct 2012 7:30 UTC

  35 points

  in reply to: Vaniver’s comment on: Want­ing to Want

  Let me try using an extended metaphor to explain my point: Remember Eliezer’s essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?

  Let’s imagine a race of Pebblesorters that’s p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.

  Now let’s suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn’t a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.

  The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn’t a prime number. They begin to persecute 11-Pebble Favorers and imprison or kill them. They declare that Sortulon Prime, the mighty Pebblesorter God that sorts stars into gigantic prime-numbered constellations in the sky, is horribly offended that some Pebblesorters favor 11 pebble piles and will banish any 11-Pebble Favorers to P-Hell, where they will be forced to sort pebbles into heaps of 8 and 9 for all eternity.

  Now let’s take a look at an individual Pebblesorter named Larry the Closet 11-Pebble Favorer. He was raised by devout 13-Pebble Favorer parents and brought up to believe that 11 isn’t a prime number. He has a second order desire to sort pebbles into prime-numbered heaps, and a first order desire to favor 11-pebble heaps. Larry is stricken by guilt that he wants to make 11-pebble heaps. He knows that 11 isn’t a prime number, but still feels a strong first order desire to sort pebbles into heaps of 11. He wishes he didn’t have that first order desire, since it obviously conflicts with his second order desire to sort pebbles into prime numbered heaps.

  Except, of course, Larry is wrong. 11 is a prime number. His first and second order desires are not in conflict. He just mistakenly thinks they are because his parents raised him to think 11 wasn’t a prime number.

  Now let’s make the metaphor explicit. Sorting pebbles into prime-numbered heaps represents Doing the Right Thing. Favoring 13-pebble heaps represents heterosexuality, favoring 11-pebble heaps represents homosexuality. Heterosexual sex and love and homosexual sex and love are both examples of The Right Thing. The people who think homosexuality is immoral are objectively mistaken about what is and isn’t moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.

  So the first and second order desires of Larry the Closet Homosexual and Larry the Closet 11-Pebble Favorer aren’t really in conflict. They just think they are because their parents convinced them to believe in falsehoods.

  I am having trouble seeing a significant difference between that and what you’ve described. Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,” and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the “serious interference” line.

  Again, I assumed that Mimi was a psychologically normal human who had normal human second order desires, like having friends and family, being healthy, doing something important with her life, challenging herself, and so on. I assumed she didn’t want to use heroin because doing so interfered with her achievement of these important second order desires.

  I suppose Mimi could be a mindless hedonist whose second order desires are somehow mistaken about what she really wants, but those weren’t the inferences I drew.

  Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,”

  Again, recall my mention of a hypothetical Heroin 2.0 in my earlier comment. It seems to me that if Heroin 2.0 was suddenly invented, and Mimi still didn’t want to use heroin, even though it no longer seriously interfered with her other important values, that she might be mistaken. Her second order desire might be a cached thought leftover from when she was addicted to Heroin 1.0 and she can safely reject it.

  But I will maintain that if Larry and Mimi are fairly psychologically normal humans, that Mimi’s second order desire to stop using heroin is an authentic and proper desire, because heroin use seriously interferes with the achievement of important goals and desires that normal humans (like Mimi, presumably) have. Larry’s second order desire, by contrast, is mistaken, because it’s based on the false belief that homosexuality is immoral. Homosexual desires do not interfere with important goals humans have. Rather, they are an important goal that humans have (love, sex, and romance), it’s just that the objective of that goal is a bit unusual (same sex instead of opposite).

  EDITED: To change some language that probably sounded too political and judgemental. The edits do not change the core thesis in any way.

• Ghatanathoah 10 May 2012 5:40 UTC

  30 points

  on: If epistemic and in­stru­men­tal ra­tio­nal­ity strongly conflict

  Yes, you probably think you care about believing truth – but isn’t it more plausible that you mainly care about thinking you like truth? Doesn’t that have a more plausible evolutionary origin than actually caring about far truth?

  Imagine I told Robin Hanson I liked the way chocolate tastes. Do you think he’d reply: “Yes, you probably think you like the taste of chocolate – but isn’t it more plausible that you mainly care about eating calorically dense foods so you can store up fat for the winter? Doesn’t that have a more plausible evolutionary origin than actually caring about the taste of chocolate?” Of course not, because that would sound silly. It’s only for abstract intellectual desires that someone can get away with a statement like that.

  If evolution “wants” you to eat calorically dense foods it doesn’t make you actually want calories, it just makes you like the way the foods taste. And if evolution “wants” you to appear to care about truth to impress people the most efficient way for it to accomplish that is to make you actually care about the truth. That way you don’t have to keep your lies straight. People don’t think they care about the truth, they actually do.

  I know that that’s Hanson’s quote, not yours, but the fact that you quote it indicates you agree with it to some extent.

Dy­ing in Many Worlds

• Ghatanathoah 1 Apr 2013 3:34 UTC

  20 points

  on: Gen­er­al­iz­ing From One Example

  I wonder if the typical mind fallacy explains some of the varying philosophical views on personal identity.

  I personally have a very strong sense of personal identity. On introspection I can definitely see that I possess certain characteristics that I consider my personal identity. I definitely think there is a “me” that persists in time.

  Of course, introspection isn’t very reliable, so I examined old home videos of me as a child, and things I had written when I was little. It wasn’t hard at all to notice many characteristics that I still possess today. The child version of me had a similar personality, quirks, interests, values, and so on. He was obviously a “younger me,” not a different person.

  However, I’ve heard other people argue that personal identity is obviously an illusion, that you aren’t the same person you were in the past. Such views seemed obviously insane to me at first, but it occurred to me that maybe other people lack the same sense of connectedness to their past self that I did. Maybe the philosophers who have argued against, or partially against personal identity (Hume, Parfit, and Giles for instance), have very weak senses of self. They mistakenly think everyone is like them and that other people are just under some sort of illusion.

  What especially disturbs me about this instance of the typical mind fallacy is that some people have taken it to mean that personal identity has no moral significance. For instance, I’ve heard arguments that individual people don’t matter, all that matters is the total quantity of pleasure, experiences, or some other fake utility function the proponent has. It seems disturbing to think a simple instance of generalizing from one example could lead to such grave moral repercussions.

• Ghatanathoah 22 Dec 2012 5:57 UTC

  17 points

  in reply to: wwa’s comment on: Dy­ing in Many Worlds

  I find this statement disturbing. This reads as if you’d really like X to be as horrible as possible to justify your preexisting decision of fighting X.

  You’re right, I phrased that badly. Let me restate it in a more rigorous fashion:

  1. I find certain moral arguments regarding the morality of making multiple versions of people to be plausible.

  2. However, when I apply those arguments to scenarios involving MWI they seem to generate results that contradict basic, common sense morality (such as “don’t kill people”)

  3. Either the moral arguments are wrong, common sense morality is wrong, or I am applying the arguments to MWI scenarios incorrectly.

  4. Regardless of which of the above is correct, it means that at the moment I am extremely uncertain of very basic moral principles.

  5. I am the sort of person who feels extreme distress at the prospect of uncertainty, especially moral uncertainty.

  6. Therefore, thinking about these things makes me feel extreme distress.

  So, to put in in a more succinct fashion, I don’t think my distress is caused by the Sunk Cost Fallacy. Rather, it is caused by an extreme dislike of moral and philosophical uncertainty. I can seen how you could interpret my initial post otherwise, however.

  In general I tend to get very worried any time my explicitly reasoned moral beliefs contradict common sense morality, for reasons PhilGoetz explains better than I can. For every one of these contradictions that is genuine moral progress there are thousands of crazy errors. In this case I suspect I’m just applying MWI improperly, but the uncertainty is very distressing to me.

  What links here?

  • Ghatanathoah's comment on Dy­ing in Many Worlds by Ghatanathoah (22 Dec 2012 6:10 UTC; 1 point)

• Ghatanathoah 11 Dec 2012 11:12 UTC

  17 points

  on: How to Avoid the Con­flict Between Fem­i­nism and Evolu­tion­ary Psy­chol­ogy?

  I believe that a large portion of people who think feminism and ev-psych conflict are making some form of the mistake Eliezer describes in The Evolutionary Cognitive Boundary.

  When someone says, “People do X to signal Y”, I tend to hear, “People do X when they consciously or subconsciously expect it to signal Y”, not, “Evolution built people to do X as an adaptation that executes given such-and-such circumstances, because in the ancestral environment, X signaled Y.”....

  ....All that should matter for saying “The parent truly cares about the child” is that the grief in the parent’s mind is cognitively real and unconditional and not even subconsciously for the sake of any ulterior motive....

  .....Of course the emotional circuitry is ultimately there for evolutionary-historical reasons. But only conscious or subconscious computations can gloom up my day; natural selection is an alien thing whose ‘decisions’ can’t be the target of my cynicism or admiration.

  To be more explicit, many feminists probably get upset at many of the ideas that ev-psych proposes because, if one does not keep the evolutionary-cognitive boundary in mind, those theories make women (and men too, come to think of it) look like calculating, manipulative sociopaths.

  For instance, if an evolutionary psychologist says “Evolution caused women to be attracted to certain types of men in order to increase the odds of them obtaining good genes and support for their children,” someone who isn’t keeping the EvCog Boundary in mind will probably hear “Women are cold, calculating, conniving monsters who manipulate men and string them along so they can get good genes for their kids and then trap men into raising them.”

  Now, that’s obvious nonsense. The vast majority of women are not manipulating anyone, they are not making some secret calculations about how to obtain good genes for their kids, and are not trying to trap men. They are just executing adaptations. The attraction they feel is totally genuine and sincere. It is natural selection that did all the cruel, amoral calculation. No one should be held personally responsible for the actions an amoral natural force took when it designed them.

  And just to be clear, I’m certainly not claiming that all women are attracted to certain types of men or anything like that. It was just the first relevant ev-psych theory that came to mind.

  It doesn’t help, of course, that there are large groups of men who are dedicated to insulting and condemning women; and that these men have realized that holding women personally responsible for the “motives” that natural selection had when it “designed” them is a great way to give their unpleasantness a scientific veneer. That’s basically what Roissy (or Heartiste, as I think he’s called now) does. For instance, that whole “cuckolding is the same as rape” nonsense of his is based on the (dead wrong) belief that people consciously desire to spread their genes.

  What links here?

  • Us­ing Evolu­tion for Mar­riage or Sex by diegocaleiro (6 May 2013 5:34 UTC; 17 points)

The Creat­ing Bob the Jerk prob­lem. Is it a real prob­lem in de­ci­sion the­ory?

[Question] How do bounded util­ity func­tions work if you are un­cer­tain how close to the bound your util­ity is?

• Ghatanathoah 24 Oct 2012 3:25 UTC

  12 points

  in reply to: PhilGoetz’s comment on: Want­ing to Want

  Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn’t really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn’t the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn’t wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.

  Mimi the Heroin Addict, by contrast, probably doesn’t want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin’s negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.

  What links here?

  • Ghatanathoah's comment on Want­ing to Want by Alicorn (25 Oct 2012 7:30 UTC; 35 points)

Is Equal­ity Really about Diminish­ing Marginal Utility?

• Ghatanathoah 24 Sep 2013 20:58 UTC

  11 points

  on: Is Kiryas Joel an Un­happy Place?

  I recently became aware of some news stories that shed some additional light on this debate:

  Ultra-Orthodox Shun Their Own for Reporting Child Sexual Abuse

  Sex abuse victim driven out of shull

  Yeshiva U sex abuse extended beyond high school for boys, probe Finds

  This is also relevant to the discussion Vladimir_M and and JoshuaZ had about whether or not the community had the ability to control social pathologies better than mainstream society (specifically it supports JoshuaZ’s position).

  My own view on the overall debate is that it doesn’t matter if Kiryas Joel is happy or not. Happiness that comes from having mistaken beliefs isn’t valuable. The majority of Ultra-orthodox Jews hold a false belief that they are giving up a normal life in order to serve a supernatural creature. Since the creature they are serving isn’t real, their lives are much, much worse than they think they are. An analogous situation might be a person who gains happiness from donating money to help starving refugees, without knowing that the refugees were made up by a con-man who is really lining his own pockets with the donations.

  This sex-abuse scandal means that the inhabitants of Kiryas Joel are even worse off than I previously thought. It’s bad enough they’re denying themselves the pleasures of mainstream out of fealty to a fiction. If they’re allowing themselves to be tortured, or to allow torturers to get away with their crimes, they are truly leading terrible lives. To use the fake refugee analogy again, suppose the donor starts mugging people to get more money to donate to the fictional refugees.

• Ghatanathoah 8 Jan 2013 5:41 UTC

  11 points

  in reply to: ctl’s comment on: Mo­ral­ity is Awesome

  But the logic that Eliezer is using is exactly the same logic that drives somebody who’s dying of a horrible disease to refuse antibiotics, because she wants to keep her body natural. And this choice is — well, it isn’t wrong, choices can’t be “wrong” — but it reflects a very fundamental sort of human bias. It’s misguided.

  Very well, let’s back up Eliezer’s argument with some hard evidence. Fortunately, lukeprog has already written a brief review of the neuroscience on this topic. The verdict? Eliezer is right. People value things other than happiness and pleasure. The idea that pleasant feelings are the sole good is an illusion created by the fact that the signals for wanting something and getting pleasure from it are comingled on the same neurons.

  So no, Eliezer is not misguided. On the contrary, the evidence is on his side. People really do value more things than just happiness. If you want more evidence consider this thought experiment Alonso Fyfe cooked up:

  Assume that you and somebody you care about (e.g., your child) are kidnapped by a mad scientist. This scientist gives you two options:

  Option 1: Your child will be taken away and tortured. However, you will be made to believe that your child is living a happy and healthy life. You will receive regular reports and even correspondence explaining how great your child’s life is. Except, they will all be fake. In fact, we will take your child to another location and spend every day peeling off his skin while soaking him in a vat of salt water, among other things.

  Option 2: Your child will be taken away, provided with paid medical insurance, an endowment to complete an education, will be hired into a good job, and will be caused to live a healthy and happy life. However, you will be made to believe that your child is suffering excruciating torture. You will be able to hear what you think are your child’s screams coming down the hallway. We will show you video of the torture. It will all be fake, of course, but you will be convinced it is real.

  Of course, after you make your choice, we will make you forget that you even had these options presented to you.

  What do you choose?

  Now, we are not going to kidnap people and make them choose. However, both theories need to explain the fact that the vast majority of parents, for example, report that, in such a situation, they would choose Option 2.

  Happiness theory seems to suggest that the agent should choose Option 1. After all, the agent will be happier receiving news (that she believes) that says that her child is living a happy and healthy life. So, if happiness is what she is after, and Option1 delivers more happiness, then Option 1 is the rational choice.

  Why do people choose Option 2?

  Because happiness theory is wrong. In fact, people do not choose happiness. They choose “making or keeping true the propositions that are the objects of our desires.” In this case, the desire in question is the desire that one’s child be healthy and happy. A person with a desire that “my child is healthy and happy” will select that option that will make or keep the proposition, “my child is healthy and happy” true. That is Option 2.

• Ghatanathoah 27 Jan 2012 21:25 UTC

  11 points

  on: The Sub­sti­tu­tion Principle

  It seems to me like the substitution principle might be an explanation for the power of framing effects. You get a different answer for a question depending on what question you end up substituting for System 1 to answer.

  That might explain the framing effect Eliezer noticed in Circular Altruism. The first time the question was asked it triggered System 1 to ask “How much would it suck if that gamble in option 2 didn’t pay off?” When the question was reframed it triggered System 1 to ask “How much does it suck that 100 people die with certainty?”

  The probably explains why political activists like to fight over framing effects so much. When people are asked the question “Would it be good for the government to do this?” System 1 spits out “Would it be good for my favorite metaphor for the government to do something vaguely analogous.” What specific metaphor System 1 gets fed would obviously have huge power in framing the question.

• Ghatanathoah 26 Oct 2012 0:29 UTC

  10 points

  in reply to: Vaniver’s comment on: Want­ing to Want

  if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I’m sure you could think of some things to write in the “pro” column for “take the pill” and in the “con” column for “don’t take the pill.”

  I don’t deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.

  However, I would maintain that the specific reason given in Alicorn’s orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.

  It’s also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it.

  I find it illuminating to compare intuitions about homosexuality to intuitions about bisexuality. If homosexual relationships were really inferior to heterosexual ones in some important way then it would make sense to encourage bisexual people to avoid homosexual relationships and focus on heterosexual ones. This seems wrong to me however, if I was giving a bisexual person relationship advice I think the good thing to do would be advise them to focus on whoever is most compatible with them regardless of sex.

  In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)

  I think you are probably right, this is proper. I think I may feel biased in favor of second order desires because right now it seems like in my current life I have difficulty preventing my first order desires from overriding them. But if I think about it, it seems like I have many first order desires I cherish and would really prefer to avoid changing.

• Ghatanathoah 24 Oct 2012 4:51 UTC

  10 points

  in reply to: Vaniver’s comment on: Want­ing to Want

  In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.

  The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results “Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal.” I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of the same sex as the desirer.

  Obviously if Larry lived in some intolerant hellhole of a country or time period it might be well to change his orientation to be bisexual or heterosexual so that he could satisfy his terminal goals of Sex and Romance without jeopardizing his terminal goals of Not Being Tortured and Killed. But that would be a second best the solution, the ideal solution would be to convince his fellows that their intolerance was unethical.

• Ghatanathoah 13 Jun 2012 7:16 UTC

  10 points

  in reply to: timtyler’s comment on: In Praise of Boredom

  Our civilisation maximises entropy—not paperclips—which hardly seems much more interesting.

  No it doesn’t. It tries to maximize fun. It might maximize entropy as a side effect, but saying that we act to maximize entropy is as ludicrous as saying cows act to maximize atmospheric methane content. You’re confusing a side-effect with the real goal.

  A constant theme I’ve noticed in your posts is that you take some (usually evolutionary) trend that is occurring in our society or history and then act as if that trend is an actual conscious goal of human beings, and life. You then exhibit confusion when someone makes a statement that their real conscious goals are in opposition to this trend, and tell them that the fact that this trend is occurring means they must really want it to occur, and that it is part of their real goals. This demonstrates rather confused thinking, both on how human minds work, an on what a “goal” really is.

  Scientists often metaphorically describe trends as having goals and “maximizing” things, you seem to have taken these metaphors excessively literally and act like these trends literally have goals and literally maximize things. Terms like “goals” and “maximization” only refer, in the literal sense, to the computations of consequentialist thinking beings (consequentialist in this case meaning a being that can forecast the future, not the moral theory). A goal is a forecast of the future a consequentialist assigns favorable utility to. Maximizing refers to a consequentialist that values a certain property in the future so much it assigns very favorable utility to increasing it as much as possible. This is the only appropriate time to literally use the terms “goal” and “maximize,” all other times are metaphorical.

  Evolution and other trends do not literally maximize anything. They have no goals. It is just sometimes useful to metaphorically pretend they do because it makes thinking about it easier for human brains, which are better at modeling other consequentialist than they are at modeling abstract trends. Any claim that evolution has a goal in a non-metaphorical sense is a blatant falsehood. Until you realize this fact you will fail to understand virtually everything that Eliezer is talking about.

  There is no good reason why this would lead to a “worthless, valueless future”—which is why Yudkowsky fails to provide one.

  Yes there is. Such a future would be boring. It is bad for things to be boring and good for things to be interesting, so such a future would be bad. And don’t ask me why boringness is bad, that’s like asking why water is H2O. You are asking Eliezer to provide some meaning of good seperate from things like truth, fun, novelty, life, etc, when he has clearly explained that there is no meaning of good seperate from those things. Those things are good, so a world where they don’t exist, or are reduced, would be bad, full stop.

• Ghatanathoah 23 May 2012 7:27 UTC

  10 points

  on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Non-positional, mutually-satisfiable values (physical luxury, for instance) Positional, zero-sum social values, such as wanting to be the alpha male or the homecoming queen

  All mutually-satisfiable values have more in common with each other than they do with any non-mutually-satisfiable values, because mutually-satisfiable values are compatible with social harmony and non-problematic utility maximization, while non- mutually-satisfiable values require eternal conflict.

  David Friedman pointed out that this isn’t correct, it’s actually it’s quite easy to make positional values mutually satisfiable:

  It seems obvious that, if one’s concern is status rather than real income, we are in a zero sum game..… Like many things that seem obvious, this one is false. It is true that my status is relative to yours. It does not, oddly enough, follow that if my status is higher than yours, yours must be lower than mine, or that if my status increases someone else’s must decrease. Status is not, in fact, a zero sum game.

  This point was originally made clear to me when I was an undergraduate at Harvard and realized that Harvard had, in at least one interesting way, the perfect social system: Everyone at the top of his own ladder. The small minority of students passionately interested in drama knew perfectly well that they were the most important people at the university; everyone else was there to provide them with an audience....

  Being a male nurse is not a terribly high status job—but that may not much matter if you are also King of the Middle Kingdom. And the status you get by being king does not reduce the status of the doctors who know that they are at the top of the medical ladder and the nurses at the bottom.

  [Emphasis mine]

  A FAI could simply make sure that everyone is a member of enough social groups that everyone has high status in some of them. Positional goals can be mutually satisficed, if one is smart enough about it. Those two types of value don’t differ as much as you seem to think they do. Positional goals just require a little more work to make implementing them conflict-free than the other type does.

  If you extract a set of consciously-believed propositions from an existing agent, then build a new agent to use those propositions in a different environment, with an “improved” logic, you can’t claim that it has the same values, since it will behave differently.

  I don’t think I agree with this. Couldn’t you take that argument further and claim that if I undergo some sort of rigorous self-improvement program in order to better achieve my goals in life, that that must mean I now have different values? In fact, you could easily say that I am behaving pointlessly because I’m not achieving my values better, I’m just changing them? It seems likely that most of the things that you are describing as values aren’t really values, they’re behaviors. I’d regard values as more “the direction in which you want to steer the world,” both in terms of your external environment and your emotional states. Behaviors are things you do, but they aren’t necessarily what you really prefer.

  I agree that a more precise and articulate definition of these terms might be needed to create a FAI, especially if human preferences are part of a network of some sort as you claim, but I do think that they cleave reality at the joints.

  I can’t really see how you can attack CEV by this route without also attacking any attempt at self-improvement by a person.

  A point I may not have made in these posts, but made in comments, is that the majority of humans today think that women should not have full rights, homosexuals should be killed or at least severely persecuted, and nerds should be given wedgies. These are not incompletely-extrapolated values that will change with more information; they are values. Opponents of gay marriage make it clear that they do not object to gay marriage based on a long-range utilitarian calculation; they directly value not allowing gays to marry. Many human values horrify most people on this list, so they shouldn’t be trying to preserve them.

  The fact that these values seem to change or weaken as people become wealthier and better educated indicates that they probably are poorly extrapolated values. Most of these people don’t really want to do these things, they just think they do because they lack the cognitive ability to see it. This is emphasized by the fact that these people, when called out on their behavior, often make up some consequentialist justification for it (if I don’t do it God will send an earthquake!)

  I’ll use an example from my own personal experience to illustrate this, when I was little (around 2-5) I thought horror movies were evil because they scared me. I didn’t want to watch horror movies or even be in the same room with a horror movie poster. I thought people should be punished for making such scary things. Then I got older and learned about freedom of speech and realized that I had no right to arrest people just because they scare me.

  Then I got even older and started reading movie reviews. I became a film connoisseur and became sick of hearing about incredible classic horror movies, but not being able to watch them because they scared me. I forced myself to sit through Halloween, A Nightmare on Elm Street, and The Grudge, and soon I was able to enjoy horror movies like a normal person.

  Not watching horror movies and punishing the people who made them were the preferences of young me. But my CEV turned out to be “Watch horror movies and reward the people who create them.” I don’t think this was random value drift, I think that I always had the potential to love horror movies and would have loved them sooner if I’d had the guts to sit down and watch them. The younger me didn’t have different terminal values, his values were just poorly extrapolated.

  I think most of the types of people you mention would be the same if they could pierce through their cloud of self-deception. I think their values are wrong and that they themselves would recognize this if they weren’t irrational. I think a CEV would extrapolate this.

  But even if I’m wrong, if there’s a Least Convenient Possible world where there are otherwise normal humans who have “kill all gays” irreversibly and directly programmed into their utility function, I don’t think a CEV of human morality would take that into account. I tend to think that, from an ethical standpoint, malicious preferences (that is, preferences where frustrating someone else’s desires is an end in itself, rather than a byproduct of competing for limited resources) deserve zero respect. I think that if a CEV took properly extrapolated human ethics it would realize this. It might not hurt to be extra careful about that when programming a CEV, however.

  What links here?

  • Ghatanathoah's comment on Hu­man val­ues differ as much as val­ues can differ by PhilGoetz (23 May 2012 8:12 UTC; -6 points)