Example 2: Suppose Mimi the Heroin Addict, living up to her unfortunate name, is a heroin addict. Obviously, as a heroin addict, she spends a lot of her time wanting heroin. But this desire is upsetting to her. She wants not to want heroin, and may take actions to stop herself from wanting heroin, such as going through rehab.
Example 3: Suppose Larry the Closet Homosexual, goodness only knows why his mother would name him that, is a closet homosexual. He has been brought up to believe that homosexuality is gross and wrong. As such, his first-order desire to exchange sexual favors with his friend Ted the Next-Door Neighbor is repulsive to him when he notices it, and he wants desperately not to have this desire.
I’m really bothered by my inability to see how to distinguish between these two classes of meta-wants. I suppose you just punt it off to your moral system, or your expected-value computations.
Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn’t really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn’t the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn’t wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.
Mimi the Heroin Addict, by contrast, probably doesn’t want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin’s negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.
In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.
The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results “Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal.” I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of the same sex as the desirer.
Obviously if Larry lived in some intolerant hellhole of a country or time period it might be well to change his orientation to be bisexual or heterosexual so that he could satisfy his terminal goals of Sex and Romance without jeopardizing his terminal goals of Not Being Tortured and Killed. But that would be a second best the solution, the ideal solution would be to convince his fellows that their intolerance was unethical.
I suppose you just punt it off to your moral system, or your expected-value computations.
I am having trouble seeing a significant difference between that and what you’ve described. Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,” and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the “serious interference” line. Similarly, the illegality of heroin and the illegality of homosexuality seem similarly comparable; perhaps Mimi should convince her fellows that their intolerance of her behavior is unethical.
Let me try using an extended metaphor to explain my point: Remember Eliezer’s essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?
Let’s imagine a race of Pebblesorters that’s p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.
Now let’s suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn’t a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.
The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn’t a prime number. They begin to persecute 11-Pebble Favorers and imprison or kill them. They declare that Sortulon Prime, the mighty Pebblesorter God that sorts stars into gigantic prime-numbered constellations in the sky, is horribly offended that some Pebblesorters favor 11 pebble piles and will banish any 11-Pebble Favorers to P-Hell, where they will be forced to sort pebbles into heaps of 8 and 9 for all eternity.
Now let’s take a look at an individual Pebblesorter named Larry the Closet 11-Pebble Favorer. He was raised by devout 13-Pebble Favorer parents and brought up to believe that 11 isn’t a prime number. He has a second order desire to sort pebbles into prime-numbered heaps, and a first order desire to favor 11-pebble heaps. Larry is stricken by guilt that he wants to make 11-pebble heaps. He knows that 11 isn’t a prime number, but still feels a strong first order desire to sort pebbles into heaps of 11. He wishes he didn’t have that first order desire, since it obviously conflicts with his second order desire to sort pebbles into prime numbered heaps.
Except, of course, Larry is wrong. 11 is a prime number. His first and second order desires are not in conflict. He just mistakenly thinks they are because his parents raised him to think 11 wasn’t a prime number.
Now let’s make the metaphor explicit. Sorting pebbles into prime-numbered heaps represents Doing the Right Thing. Favoring 13-pebble heaps represents heterosexuality, favoring 11-pebble heaps represents homosexuality. Heterosexual sex and love and homosexual sex and love are both examples of The Right Thing. The people who think homosexuality is immoral are objectively mistaken about what is and isn’t moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.
So the first and second order desires of Larry the Closet Homosexual and Larry the Closet 11-Pebble Favorer aren’t really in conflict. They just think they are because their parents convinced them to believe in falsehoods.
I am having trouble seeing a significant difference between that and what you’ve described. Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,” and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the “serious interference” line.
Again, I assumed that Mimi was a psychologically normal human who had normal human second order desires, like having friends and family, being healthy, doing something important with her life, challenging herself, and so on. I assumed she didn’t want to use heroin because doing so interfered with her achievement of these important second order desires.
I suppose Mimi could be a mindless hedonist whose second order desires are somehow mistaken about what she really wants, but those weren’t the inferences I drew.
Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,”
Again, recall my mention of a hypothetical Heroin 2.0 in my earlier comment. It seems to me that if Heroin 2.0 was suddenly invented, and Mimi still didn’t want to use heroin, even though it no longer seriously interfered with her other important values, that she might be mistaken. Her second order desire might be a cached thought leftover from when she was addicted to Heroin 1.0 and she can safely reject it.
But I will maintain that if Larry and Mimi are fairly psychologically normal humans, that Mimi’s second order desire to stop using heroin is an authentic and proper desire, because heroin use seriously interferes with the achievement of important goals and desires that normal humans (like Mimi, presumably) have. Larry’s second order desire, by contrast, is mistaken, because it’s based on the false belief that homosexuality is immoral. Homosexual desires do not interfere with important goals humans have. Rather, they are an important goal that humans have (love, sex, and romance), it’s just that the objective of that goal is a bit unusual (same sex instead of opposite).
EDITED: To change some language that probably sounded too political and judgemental. The edits do not change the core thesis in any way.
We should point people to this whenever they’re like “What’s special about Less Wrong?” and we can be like “Okay, first, guess how Less Wrong would discuss a reluctant Christian homosexual. Made the prediction? Good, now click this link.”
First, I would like to make one thing clear: I have absolutely nothing against homosexuals and in fact qualify as queer because my attractions transcend gender entirely. I call my orientation “sapiosexual” because it is minds that I am sexually attracted to, and good character, never mind the housing.
Stops at “pigheaded jerks”
downvotes
You know where this is going, oh yes, I am going right to fundamental attribution error and political mindkill.
The parents are deemed “pigheaded jerks”—a perception of their personality.
Larry the homosexual, convinced by the exact same reasoning, is given something subtly different—an attack on his behavior—“he gullibly believed them” and you continue with “They (the Larrys) just think they are because their parents fed them a load of crap.” attributing his belief to the situation that Larry is in.
Do you think Larry’s grandparents didn’t teach Larry’s parents the same thing? And that Larry’s great grandparents didn’t teach it to Larry’s grandparents?
I upvoted despite this. If you overlook that one problem, everything else is gold. That single flawed sentence does not effect the awesome of the other 14 paragraphs, as it does not contribute to the conclusion.
I am charmed by your polite acknowledgement of the flaw and am happy to see that this has been updated. Thanks for letting me know that pointing it out was useful. :)
I think you’re looking at this discussion from the wrong angle. The question is, “how do we differentiate first-order wants that trump second-order wants from second-order wants that trump first-order wants?” Here, the order only refers to the psychological location of the desire: to use Freudian terms, the first order desires originate in the id and the second order desires originate in the superego.
In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)
Some cases are easier than others- in the case of Sally, who wants to commit suicide but wants to not want to commit suicide, I would generally recommend methods of effective treatment for suicidal tendencies, not the alternative. But you should be able to recognize that the decision could be difficult, at least for some alteration of the parameters, and is the alteration is significant enough it could swing the other way.
There is also another factor which clouds the analysis, which is that the ego has to weigh the costs of altering, suppressing, or foregoing one of the desires. It could be that Larry has a twin brother, Harry, who is not homosexual, and that Harry is genuinely happier that Larry is, and that Larry would genuinely prefer being Harry to being himself; he’s not mistaken about his second-order want.
However, the plan to be (or pretend to be) straight is much more costly and less likely to succeed than the plan to stop wanting to be straight, and that difference in costs might be high enough to determine the ego’s decision. Again, it should be possible to imagine realistic cases in which the decision would swing the other way. (Related.)
It’s also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it. (Note Alicorn’s response. Given that homosexuality is immutable, then plans to change homosexuals are unlikely to succeed, and they might as well make the best of their situation. But I hope it’s clear that, at its root, this is a statement about engineering reality, not moral principles- if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I’m sure you could think of some things to write in the “pro” column for “take the pill” and in the “con” column for “don’t take the pill.”
Why I said this is worth considering is that, as should be unsurprising, two wants conflict. Often, we don’t expect the engineering reality to change. Male homosexuality is likely to be immutable for the lifetimes of the ones that are currently alive, and it’s more emotionally satisfying to declare that homosexual desires don’t conflict with important goals than reflect on the tradeoffs that homosexuals face that heterosexuals don’t. Doing so, however, requires a sort of willful blindness, which may or may not be worth the reward gained by engaging in it.
if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I’m sure you could think of some things to write in the “pro” column for “take the pill” and in the “con” column for “don’t take the pill.”
I don’t deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.
However, I would maintain that the specific reason given in Alicorn’s orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.
It’s also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it.
I find it illuminating to compare intuitions about homosexuality to intuitions about bisexuality. If homosexual relationships were really inferior to heterosexual ones in some important way then it would make sense to encourage bisexual people to avoid homosexual relationships and focus on heterosexual ones. This seems wrong to me however, if I was giving a bisexual person relationship advice I think the good thing to do would be advise them to focus on whoever is most compatible with them regardless of sex.
In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)
I think you are probably right, this is proper. I think I may feel biased in favor of second order desires because right now it seems like in my current life I have difficulty preventing my first order desires from overriding them. But if I think about it, it seems like I have many first order desires I cherish and would really prefer to avoid changing.
While the Freudian description is accurate relative to sources, I struggle to order them. I believe it is an accumulated weighting that makes one thought dominate another. We are indeed born with a great deal of innate behavioral weighting. As we learn, we strengthen some paths and create new paths for new concepts. The original behaviors (fight or flight, etc.) remain.
Based on this known process, I conjecture that experiences have an effect on the weighting of concepts. This weighting sub-utility is a determining factor in how much impact a concept has on our actions. When we discover fire burns our skin, we don’t need to repeat the experience very often to weigh fire heavily as something we don’t want touching our skin.
If we constantly hear, “blonde people are dumb,” each repetition increases the weight of this concept. Upon encountering an intelligent blond named Sandy, the weighting of the concept is decreased and we create a new pattern for “Sandy is intelligent” that attaches to “Sandy is a person” and “Sandy is blonde.” If we encounter Sandy frequently, or observe many intelligent blonde people, the weighting of the “blonde people are dumb” concept is continually reduced.
Coincidentally, I believe this is the motivation behind why religious leaders urge their followers to attend services regularly, even if subconsciously. The service maintains or increases weighting on the set of religious concepts, as well as related concepts such as peer pressure, offsetting any weighting loss between services. The depth of conviction to a religion can potentially be correlated with frequency of religious events. But I digress.
Eventually, the impact of the concept “blonde people are dumb” on decisions becomes insignificant. During this time, each encounter strengthens the Sandy pattern or creates new patterns for blondes. At some level of weighting for the “intelligent” and “blonde” concepts associated to people, our brain economizes by creating a “blond people are intelligent” concept. Variations of this basic model is generally how beliefs are created and the weights of beliefs are adjusted.
As with fire, we are extremely averse to incongruity. We have a fundamental drive to integrate our experiences into a cohesive continuum. Something akin to adrenaline is released when we encounter incongruity, driving us to find a way to resolve the conflicting concepts. If we can’t find a factual explanation, we rationalize one in order to return to balanced thoughts.
When we make a choice of something over other things, we begin to consider the most heavily weighted concepts that are invoked based on the given situation. We work down the weighting until we reach a point where a single concept outweighs all other competing concepts by an acceptable amount.
In some situations, we don’t have to make many comparisons due to the invocation of very heavily weighted concepts, such as when a car is speeding towards us while we’re standing in the roadway. In other situations, we make numerous comparisons that yield no clear dominant concept and can only make a decision after expanding our choice of concepts.
This model is consistent with human behavior. It helps to explain why people do what they do. It is important to realize that this model applies no division of concepts into classes. It uses a fluid ordering system. It has transient terminal goals based on perceived situational considerations. Most importantly, it bounds the recursion requirements. As the situation changes, the set of applicable concepts to consider changes, resetting the core algorithm.
From what I’ve heard, the typical response to believing that blond people are dumb and observing that blond Sandy is intelligent is to believe that Sandy is an exception, but blond people are dumb.
Most people are very attached to their generalizations.
Quite right about attachment. It may take quite a few exceptions before it is no longer an exception. Particularly if the original concept is regularly reinforced by peers or other sources. I would expect exceptions to get a bit more weight because they are novel, but no so much as to offset higher levels of reinforcement.
Your example does an exemplery job of explaining your viewpoint on Larry’s situation. To explain the presumed viewpoint of Larry’s parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9.
The people who think homosexuality is immoral are objectively mistaken about what is and isn’t moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.
How do you define objective morality? I’ve heard of several possible definitions, most of which conflict with each other, so I’m a little curious as to which one you’ve selected.
To explain the presumed viewpoint of Larry’s parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9.
I’m not sure I understand you, do you mean that a more precise description of Larry’s parent’s viewpoint is that the Pebblesorter versions of them think 11 and 9 are the same numbers? Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument.
How do you define objective morality?
I define morality as being a catch-all term to describe what are commonly referred to as the “good things in life,” love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things. In other words, “good” and “right” are synonyms. Morality is objective because we can objectively determine whether people are happy, being treated fairly, getting what they want out of life, etc. In Larry’s case having a relationship with Ted-the-Next-Door neighbor would be the morally right thing to do because it would increase the amount of love, happiness, people-getting what they want, etc. in the world.
I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism. That is, they believe that moral knowledge is intrinsically motivating, simply knowing something is right motivates someone to do it. They then conclude that since intrinsically motivating knowledge doesn’t seem to exist, that morality must be subjective.
I am a motivational externalist, so I do not buy this argument. I believe that that people are motivated to act morally by our conscience and moral emotions (i.e. compassion, sympathy). If someone has no motivation to act to increase the “good things in life” that doesn’t mean morality is subjective, that simply means that they are a bad person. People who lack moral emotions exist in real life, and they seem to lack any desire to act morally at all, unless you threaten to punish them if they don’t.
The idea of intrinsically motivating knowledge is pretty scary if you think about it. What if it motivated you to kill people? Or what if it made you worship Darkseid? The Anti-Life equation from Final Crisis works pretty much exactly the way motivational internalists think moral knowledge does, except that instead of motivating people to care about others and treat people well, it instead motivates them to serve evil pagan gods from outer space.
Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument.
Yes, exactly. Larry’s parents’ do not believe that they are mistaken, and are not easily proved mistaken.
I define morality as being a catch-all term to describe what are commonly referred to as the “good things in life,” love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things.
That’s a good definition, and it avoids most of the obvious traps. A bit vague, though. Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative… except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I’d still think that leaving the child locked away in the dark is a wrong thing. (This can also lead to Pascal’s Mugging, as an edge case)
I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism.
In my case, it’s because every attempt I’ve seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche? There are some bad things in the dark corners of the human psyche. Take whatever action is most likely to transform the world into a paradise? Doesn’t usually work, because we don’t know enough to always select the correct actions. Do unto others as you would have them do unto you? That’s a very nice one—but not if Bob the Masochist tries to apply it.
Of course, subjective morality is no better—and is often worse (mainly because a society in general can reap certain benefits from a shared idea of morality).
What does seem to work is to pick a society whose inhabitants seem happy and fulfilled, and trying to use whatever rules they use. The trouble with that is that it’s kludgy, uncertain, and could often do with improvement (though it’s been improved often enough in human history that many—not all, but many—obvious ‘improvements’ aren’t in practice).
Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative… except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I’d still think that leaving the child locked away in the dark is a wrong thing.
Aside from its obvious artificiality, and despite the fact that all our instincts cry out against it, it’s not at all clear to me that there are any really good reasons to reject the Omelasian solution. This is of course a fantastically controversial position (just look at the response to Torture vs. Dust Specks, which might be viewed as an updated and reframed version of the central notion of The Ones Who Walk Away From Omelas), but it nonetheless seems to be a more or less straightforward consequence of most versions of consequential ethics.
As a matter of fact, I’m inclined to view Omelas as something between an intuition pump and a full-blown cognitive exploit: a scenario designed to leverage our ethical heuristics (which are well-adapted to small-scale social groups, but rather less well adapted to exotic large-scale social engineering) in order to discredit a viewpoint which should rightfully stand or fall on pragmatic grounds. A tortured child is something that hardly anyone can be expected to think straight through, and trotting one out in full knowledge of this fact in order to make a point upsets me.
Aside from its obvious artificiality, and despite the fact that all our instincts cry out against it, it’s not at all clear to me that there are any really good reasons to reject the Omelasian solution.
There’s a real world analogue to Omelas. The UK (like other countries) has a child protection system, intended to minimize abuse & neglect of children. The state workers (health visitors, social workers, police officers, hospital staff, etc.) who play roles in the system can try to intervene when the apparent risk of harm to a child reaches some threshold.
If the threshold is too low, the system gratuitously interferes with families’ autonomy, preventing them from living their lives in peace. If the threshold is too high, the system fails to do its job of preventing guardians from tormenting or killing their children. Realistically, a trade-off is inevitable, and under any politically feasible threshold “some children will die to preserve the freedom of others”, as the sociologist Robert Dingwall put it. So the UK’s child protection system takes the Omelasian route.
The real life situation is less black & white than Omleas, but it looks like the same basic trade-off in a non-artificial setting. I wonder whether people’s intuitions about Omelas align with their intuitions about the real life child protection trade-off (and indeed whether both align with society’s revealed collective preference).
I’d agree that these are consequentially similar, but I don’t think they’re psychologically similar at all. There’s an element of exploitation in Omelas that isn’t present in social services: state workers are positioned as protecting children from an evil unrelated to the state, while Omelas is cast as willfully perpetrating an evil in order to ensure its own prosperity. People tend to think of moral culpability in terms of blame, and although some blame might attach itself to social workers for failing to stop abuses that they might prevent with more intrusive intervention thresholds, it’s much diluted by the vastly more viscerally appalling culpability carried by actual abusers. Omelas offers no subjects for condemnation other than the state apparatus and the citizens supporting it.
On top of that, intrusive child protection services have very salient failings: most parents (and most children) would find government intrusion into their family lives extremely unpleasant, unpleasant enough to fear and take political action to avoid. Meanwhile, the consequences of no longer torturing Omelas’ sacrificial lamb are unspecified and thus about as far from salient (re-entrant?) as it’s possible to get. Even in a hypothetical fully specified Omelas where we could point to a chain of effects, I’d expect that chain to be a lot longer and harder to follow, and its endpoints hence less emotionally weighty.
There’s an element of exploitation in Omelas that isn’t present in social services: state workers are positioned as protecting children from an evil unrelated to the state, while Omelas is cast as willfully perpetrating an evil in order to ensure its own prosperity.…
...Omelas offers no subjects for condemnation other than the state apparatus and the citizens supporting it.
Link to me mentioning both Omelas and another “eternally tortured child” short story, SCP-231 (potentially highly distressing so I’m not hotlinking it), as an intuition pump against Mencius Moldbug’s “Patchwork” proposal (the “strong”/”total” vision of Patchwork, with absolute security of sovereigns) along very similar lines of analogy, over in the Unqualified Reservations comments.
There are many, many things in the similarly exploitative franchise known as “Real Life” that also appear to be crafted as “torture & horror porn”. So I don’t see the problem with linking to a fictionalized version.
I dare say that any story without elements that induce horror and revulsion in a reader would be an inadequate source of intuition for considering the most shocking aspects of our own world… or the ethics of knowingly creating a system which offers absolute security indiscriminately to those who would create such nightmares and those who’d seek to prevent them.
Example of a victim testimony 2: - all of the above, except even more outspoken descriptions of the author’s mental anguish. (NSFanywhere. The main blog has… images… that are more gore than extreme porn; don’t look unless you’re massively desensitized.)
...Gurfr cubgbf nyy rkcerff gur fvqr bs zlfrys V fgehttyr jvgu rirel qnl. Guvf vf gur fvqr bs zr gung yrnearq jung frk vf guebhtu encr. Guvf vf gur fvqr bs zr gung gevrq gb pbzzvg fhvpvqr sbe gur svefg gvzr jura V jnf frira lrnef byq. Guvf vf gur cneg bs zr gung V srne jvyy arire urny. Vgf orra fb znal lrnef ohg abg n qnl tbrf ol jurer V qba’g guvax nobhg jung jnf qbar gb zr...
Want some amnesiacs yet? You might be able to forget those stories faster if you don’t think about the fact that something similar must be happening somewhere in your country, probably in your city, at this very moment. Oops, too late!
(Again, sorry for the confrontational tone and such—I wanted to hammer home the point that sometimes it’s the violently emotional reaction to an objectively terrible problem that would be true to your desires, and trying to stay “detached” and “reasonable” would be self-deception. See: deathism.)
(Again, sorry for the confrontational tone and such—I wanted to hammer home the point that sometimes it’s the violently emotional reaction to an objectively terrible problem that would be true to your desires, and trying to stay “detached” and “reasonable” would be self-deception. See: deathism.)
Indeed. But in the context of the discussion the story primes you to live one kind of horror and not another when making trade offs between the two. This is why I objected to it.
I struggle with conceiving wanting to want, or decision making in general, as a tiered model. There are a great many factors that modify the ordering and intensity of utility functions. When human neurons fire they trigger multiple concurrent paths leading to a set of utility functions. Not all of the utilities are logic-related.
I posit that our ability to process and reason is due to this pattern ability and any model that will approximate human intelligence will need to be more complex than a simple linear layer model. The balance of numerous interactive utilities combine to inform decision making. A multiobjective optimization model, such as PIBEA, is required.
I’m new to LW, so I can’t open threads just yet. I’m hoping to find some discussions around evolutionary models and solution sets relative to rational decision processing.
I sort of agree with your comment, but you should’ve probably posted it as a reply to the main post, not to this specific comment of mine, as the topics seem to be somewhat apart.
(BTW you can open threads in Discussion just fine seeing as you’re at 10 karma—go ahead!)
Disagree. Do look, even if it hurts. Especially if it hurts a lot (if it doesn’t hurt at all, there’s a lot more wrong to fix first). Update your model of reality, avoid availability and (dis)confirmation bias. Face reality head-on. That kind of thing.
All due respect, but I don’t think there is any imperative to view material that will cause serious psychological harm. If the price of knowledge is your right hand you are not required to pay it. The most relevant example would be showing hardcore pornography to small children—it may be knowledge of a kind, but it does more harm than good.
EDIT: Of course, I read them anyway. Very, ah … deep.
I only said that in regards to the many photos of staged degradation, multilation, self-harm, etc on the second blog—they’re cringe-inducing yet presumably taken in a consensual way (for a loose definition of consent), and thus have a worse “unpleasantness-to-facing-reality” ratio than the textual descriptions of real abuse & its consequences. I quite agree that reading those shocking descriptions is epistemologically and morally imperative.
Omelas is a cognitive exploit, yes. That’s really the point—it forces people to consider how appropriate their heuristics really are. Some people would make Omelas if they could; some wouldn’t, for the sake of the one child. A firm preference for either possibility can be controversial, partially because there are good reasons for both states and partially because different ethical heuristics get levered in different directions. (A heuristic that compares the number of people helped vs. the number hurt will pull one way; a heuristic that says “no torture” will pull the other way).
Personally, on reading the story, I decided immediately that not only would I not walk away from Omelas (which solves nothing anyway,) I was fully in favor of the building of Omelases, provided that even more efficient methods of producing prosperity were not forthcoming.
The prevention of dust specks may vanish into nothing in my intuitive utility calculations, but it immediately hits me that a single tortured child is peanuts besides the cumulative mass of suffering that goes on in our world all the time. With a hundred thousand dollars or so to the right charity, you could probably prevent a lot more disutility than that of the tortured child. If for that money I could either save one child from that fate, or create a city like Omelas minus the sacrifice, then it seems obvious to me that creating the city is a better bargain.
That’s true, but I think that human values are so complex that any attempt to compress morality into one sentence is pretty much obligated to be vague.
Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative… except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests.
One rather obvious rejoinder is that there are currently hundreds, if not thousands of children who are in the same state as the unfortunate Omelasian right now in real life, so reducing the number to just one child would be a huge improvement. But you are right that even one seems too many.
A more robust possibility might be to add “equality” to the list of the “good things in life.” If you do that then Omelas might be morally suboptimal because the vast inequality between the child and the rest of the inhabitants might overwhelm the achievement of the other positive values. Now, valuing equality for its own sake might add other problems, but these could probably be avoided if you were sufficiently precise and rigorous in defining equality.
In my case, it’s because every attempt I’ve seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche?
I think the best explanation I’ve seem is something like the metaethics Eliezer espouses, which is (if I understand them correctly), that morality is a series of internally consistent concepts related to achieving what I called “the goods things in life,” and that human beings (those who are not sociopaths anyway) care a lot about these concepts of wellbeing and want to follow and fulfill them.
In other words, morality is like mathematics in some ways, it generates consistent answers(on the topic of people’s wellbeing) that are objectively correct. But it is not like the Anti-Life Equation because it is not intrinsically motivating. Humans care about morality because of our consciences and our positive emotions, not because it is universally compelling.
To put it another way, I think that if you were to give a superintelligent paperclipper a detailed description of human moral concepts and offered to help it make some more paperclips if it elucidated these concepts for you, that it would probably generate a lot of morally correct answers. It would feel no motivation to obey these answers of course, since it doesn’t care about morality, it cares about making paperclips.
This is a little like morality being “embedded in the human psyche” in the sense that the desire to care about morality is certainly embedded in their somewhere (probably in the part we label “conscience”). But it is also objective in the sense that moral concepts are internally consistent independent of the desires of the mind. To use the Pebblesorter metaphor again, caring about sorting pebbles into prime numbered heaps is “embedded in the Pebblesorter psyche,” but which numbers are prime is objective.
There are some bad things in the dark corners of the human psyche.
That’s certainly true, but that simply means that humans are capably of caring about other things besides morality, and these other things that people sometimes care about can be pretty bad. This obviously makes moral reasoning a lot harder, since it’s possible that one of your darker urges might be masquerading as a moral judgement. But that just means that moral reasoning is really hard to do, it doesn’t mean that it’s wrong in principle.
That’s true, but I think that human values are so complex that any attempt to compress morality into one sentence is pretty much obligated to be vague.
Vague or flawed. Given those options, I think I’d prefer vague.
One rather obvious rejoinder is that there are currently hundreds, if not thousands of children who are in the same state as the unfortunate Omelasian right now in real life, so reducing the number to just one child would be a huge improvement. But you are right that even one seems too many.
I agree completely. If I had any idea how Omelas worked, I might be tempted to try seeing if any of those ideas could be used to improve current societies.
A more robust possibility might be to add “equality” to the list of the “good things in life.” If you do that then Omelas might be morally suboptimal because the vast inequality between the child and the rest of the inhabitants might overwhelm the achievement of the other positive values. Now, valuing equality for its own sake might add other problems, but these could probably be avoided if you were sufficiently precise and rigorous in defining equality.
Hmmm. To avoid Omelas, equality would have to be fairly heavily weighted; any finite weighting given to equality, however, will simply mean that Omelas is only possible given a sufficiently large population (by balancing the cost of the inequality with the extra happiness of the extra inhabitants).
Personally, I think that valuing equality itself is a good idea, if mixed in with a suitable set of other values; one possible failure mode for overvaluing equality is an equality of wretchedness, a state of “we’re all equal because we all have nothing and no hope” (this is counteracted by providing suitable weights for the other good things, like happiness and the freedom to try to achieve goals (but what about the goal of vengeance? For a real slight? An imagined slight?)).
An example of a society failing due to an over-reliance on equality, is France shortly after the French Revolution.
I think the best explanation I’ve seem is something like the metaethics Eliezer espouses
I think that I should read through that entire sequence in the near future. I’d like to see his take on metaethics.
In other words, morality is like mathematics in some ways, it generates consistent answers(on the topic of people’s wellbeing) that are objectively correct.
Huh. I think we’re defining ‘morality’ slightly differently here.
My definition of ‘morality’ would be ‘a set of rules, decided by some system, such that one can feed in a given action and (usually) get out whether that action was a good or a bad action’. Implicit in that definition is the idea that two people may disagree on what those rules actually are—that there might be better or worse moralities, and that therefore the answers given by a randomly chosen morality need not be objectively correct.
To take an example; certain ancient cultures may have had the belief that human sacrifice was necessary, on Midwinter’s Day, to persuade summer to come back and let the crops grow. In such a culture, strapping someone down and killing them in a particularly painful way may have been considered the right thing to do; and a member of that society would argue for it on the basis that the tribe needs the crops to grow next year (and, if selected, might even walk up voluntarily to be killed). In his morality, these annual deaths are a good thing, because they make the crops grow; in my morality, these annual deaths are a bad thing, and moreover, they don’t make the crops grow.
For what it’s worth, I do agree with you that getting the result out of a moral system does not, in and of itself, force an intrinsic motivation to follow that course of action; people can be trained from a young age to feel that motivation, and many people are, but there’s really no reason to assume that it is always there.
If there is an objectively correct morality, that can apply to all situations, then I don’t know what it is—my current system of morality (based heavily on the biblical principal of ‘Love thy neighbour’) covers many situations, but is not good at the average villain’s sadistic choice (where I can save the lives of group A or group B but not both)
There are some bad things in the dark corners of the human psyche.
That’s certainly true, but that simply means that humans are capably of caring about other things besides morality, and these other things that people sometimes care about can be pretty bad.
Hmmm. A lot of the darkness in the human psyche can be explained in this manner; but I’d think that there are other parts which cannot be explained in this way (when a person goes out of his way to hurt someone that he’ll likely never see again, for example). A lot of these, I’d think, are attributable to a lack of empathy; a person who sees other people as non-people (or as Not True People, for some self-including definition of True People).
To avoid Omelas, equality would have to be fairly heavily weighted
I think a possible solution would be to have equality and the other values have diminishing returns relative to each other. So in a society with a lot of other good things there is a great obligation to increase equality, whereas in a society with lots of suffering people it’s more important to do whatever it takes raise the general level of happiness and not worry as much about equality. So a place as wondrous as Omelas would have a great obligation to help the child.
one possible failure mode for overvaluing equality is an equality of wretchedness, a state of “we’re all equal because we all have nothing and no hope”
I think one possible way to frame equality to avoid this is to imagine, metaphorically, that positive things give a society “morality points” and negative things give it “negative morality points.” Then have it so that a positive deed that also decreases equality gets “extra points,” while a negative deed that also exasperates inequality gets “extra negative points.” So in other words, helping the rich isn’t bad, it’s just much less good than helping the poor.
This also avoids another failure mode: Imagine an action that hurts every single person in the world, and hurts the rich 10 times as much as it hurts the poor. Such an action would increase equality, but praising it seems insane. Under the system I proposed such an action would still count as “bad,” though it would be a bit less bad than a bad action that also increased inequality.
Huh. I think we’re defining ‘morality’ slightly differently here.
My definition of ‘morality’ would be ‘a set of rules, decided by some system, such that one can feed in a given action and (usually) get out whether that action was a good or a bad action’.
I don’t think that’s that different from what I’m saying, I may be explaining it poorly. I do think that morality is essentially like a set of rules or an equation that one uses to evaluate actions. And I consider it objective in that the same equation should produce the same result each time an identical action is fed into it, regardless of what entity is doing the feeding. Then it is up to our moral emotions to motivate us to take actions the equation would label as “good.”
Describing it like that sounds a bit clinical though, so I’d like to emphasize that moral rules and equations, are ultimately about people’s wellbeing and increasing the good things in life. If you feed an action that improves these values into a rule-set and it comes out labelled “bad” then those rules probably don’t even deserve to be called morality, they are some other completely different concept.
Implicit in that definition is the idea that two people may disagree on what those rules actually are—that there might be better or worse moralities, and that therefore the answers given by a randomly chosen morality need not be objectively correct.
This relates to Eliezer’s metaethics again, he basically describes morality as an equation or “fixed computation” related to wellbeing that is so complex that it’s impossible to wrap your mind about it, so you have to work in approximations. So what you would label a “better” morality is one that more closely resembles the “ideal equation.”
To take an example; certain ancient cultures may have had the belief that human sacrifice was necessary, on Midwinter’s Day, to persuade summer to come back and let the crops grow.
It seems to me that this is more a disagreement about certain facts of nature than about morality per se. It seems to me that if there really was some sort of malevolent supernatural entity that wouldn’t let summer come unless you made sacrifices to it, and it was impossible to stop such an entity, that sacrificing to it might be the only option left. If the choice is “everyone dies of starvation” vs. “one person dies from being sacrificed, everyone else lives” it seems like any worthwhile set of moral rules would label the second option as the better one (though it would not be nearly as good as somehow stopping the entity). The reason that sacrificing people is bad is because such entities do not exist, so such a sacrifice tortures someone, but doesn’t save anyone elses’ life.
If there is an objectively correct morality, that can apply to all situations, then I don’t know what it is
I think the problem is that an objectively correct set of moral rules that could perfectly evaluate any situation would be so complicated no one would be able to use it effectively. Even if we obtained such a system we would have to use crude approximations until we managed to get a supercomputer big enough to do the calculations in a timely manner.
A lot of these, I’d think, are attributable to a lack of empathy; a person who sees other people as non-people
I count empathy as one of the “moral emotions” that motivates people to act morally. So a lack of empathy would be a type of lack of motivation towards moral behavior.
I think a possible solution would be to have equality and the other values have diminishing returns relative to each other.
That seems to work very well. So the ethical weight of a factor can be proportional to the reciprocal thereof (perhaps with a sign change). Then, for any number of people, there is a maximum happiness-factor that the equation can produce.
So. This can be used to make an equation that makes Omelas bad for any sized population. But not everyone agrees that Omelas is bad in the first place; so is that necessarily an improvement to your ethical equation?
I think one possible way to frame equality to avoid this is to imagine, metaphorically, that positive things give a society “morality points” and negative things give it “negative morality points.” Then have it so that a positive deed that also decreases equality gets “extra points,” while a negative deed that also exasperates inequality gets “extra negative points.” So in other words, helping the rich isn’t bad, it’s just much less good than helping the poor.
This also avoids another failure mode: Imagine an action that hurts every single person in the world, and hurts the rich 10 times as much as it hurts the poor. Such an action would increase equality, but praising it seems insane. Under the system I proposed such an action would still count as “bad,” though it would be a bit less bad than a bad action that also increased inequality.
That failure mode can also be dealt with by combining equality with other factors, such as not being hurt. (The relative weightings assigned to these factors would be important, of course).
I don’t think that’s that different from what I’m saying, I may be explaining it poorly. I do think that morality is essentially like a set of rules or an equation that one uses to evaluate actions. And I consider it objective in that the same equation should produce the same result each time an identical action is fed into it, regardless of what entity is doing the feeding. Then it is up to our moral emotions to motivate us to take actions the equation would label as “good.”
That seems like a reasonable definition; my point is that not everyone uses the same equation.
It seems to me that this is more a disagreement about certain facts of nature than about morality per se.
Hmmm. You’re right—that was a bad example. (I don’t know if you’re familiar with the Chanur series, by C. J. Cherryh? I ask because my first thought for a better example came straight out of there—she does a god job of presenting alien moralities)
Let me provide a better one. Consider Marvin, and Fred. Marvin’s moral system considers the total benefit to the world of every action; but he tends to weight actions in favour of himself, because he knows that in the future, he will always choose to do the right thing (by his morality) and thus deserves ties broken in his favour.
Fred’s moral system entirely discounts any benefits to himself. He knows that most people are biased to themselves, and does this in an attempt to reduce the bias (he goes so far as to be biased in the opposite direction).
Both of them get into a war. Both end up in the following situation:
Trapped in a bunker, together with one allied soldier (a stranger, but on the same side). An enemy manages to throw a grenade in. The grenade will kill both of them, unless someone leaps on top of it, in which case it will only kill that one.
Fred leaps on top of the grenade. His morality values the life of the stranger over his own, and he thus acts to save the stranger first.
Marvin throws the stranger onto the grenade. His morality values his own life over a stranger who might, with non-trivial probability, be a truly villainous person.
Here we have two different moralities, leading to two different results, in the same situation.
I think the problem is that an objectively correct set of moral rules that could perfectly evaluate any situation would be so complicated no one would be able to use it effectively. Even if we obtained such a system we would have to use crude approximations until we managed to get a supercomputer big enough to do the calculations in a timely manner.
That is worth keeping in mind. Of course, if such a system is found, we could feed in dozens of general situations in advance—and if in a tough situation, then after resolving it one way or another, we could feed that situation into the computer and find out for future reference which course of action was correct (that eliminates a lot of the time constraint).
That seems like a reasonable definition; my point is that not everyone uses the same equation.
That’s true, the question is, how often is this because people have totally different values, and how often is it that they have extremely similar “ideal equations,” but different “approximations” of what they think that equation is. I think for sociopaths, and other people with harmful ego-syntonic mental disorders it’s probably the former, but its more often the later for normal people.
Eliezer has argued that it is confusing and misleading to use the word “morality” to refer to codes of behavior entities possess that have nothing to do with improving people’s wellbeing, making the world a happier, fairer, freer place, and similar concepts. He argues that creatures like the Pebblesorters do not care about morality at all, they care about sorting pebbles and calling sorting pebbles a type of “morality” confuses two separate concepts.
Let me provide a better one. Consider Marvin, and Fred.
It sounds to me like Fred and Marvin both care about achieving similar moral objectives, but have different ideas about how to go about it. I’d say that again, which moral code is better can only be determined by trying to figure out which one actually does a better job of achieving moral goals. “Moral progress” can be regarded as finding better and better heuristics to achieve those moral goals, and finding a closer representation of the ideal equation.
Again, I think I agree with Eliezer that a truly alien code of behavior, like that exhibited by sociopaths, and really inhuman aliens like the Pebblesorters or paperclippers, should maybe be referred to by some word other than morality. This is because since the word “morality” usually refers to doing things like making the world a happier place and increasing the positive things in life. So if we refer to the behavior code of a creature that cares nothing for doing those things as “morality,” we will give the subconscious impression that that creature really does care about doing good and simply disagrees about how to go about it. This isn’t correct, a sociopaths and paperclippers doesn’t care about other people at all, so we shouldn’t give the impression they do.
I am less sure about whether the term “morality” should be used to refer to the behavior codes of aliens that care about some of the same positive things that normal humans do, but also differ in important ways, like the Babyeaters and Super-Happy-People. Maybe we could call it “semi-morality?”
(I don’t know if you’re familiar with the Chanur series, by C. J. Cherryh? I ask because my first thought for a better example came straight out of there—she does a god job of presenting alien moralities)
Sorry, the only Cherryh I’ve read is “The Scapegoat.” I thought it gave a good impression of how alien values would look to humans, but wish it had given some more ideas about what it was that made elves think so differently.
That seems like a reasonable definition; my point is that not everyone uses the same equation.
That’s true, the question is, how often is this because people have totally different values, and how often is it that they have extremely similar “ideal equations,” but different “approximations” of what they think that equation is. I think for sociopaths, and other people with harmful ego-syntonic mental disorders it’s probably the former, but its more often the later for normal people.
I’d say sometimes A, and sometimes B. But I think that’s true even in the absence of mental disorders; I don’t think that the “ideal equation” necessarily sits somewhere hidden in the human psyche.
It sounds to me like Fred and Marvin both care about achieving similar moral objectives, but have different ideas about how to go about it. I’d say that again, which moral code is better can only be determined by trying to figure out which one actually does a better job of achieving moral goals. “Moral progress” can be regarded as finding better and better heuristics to achieve those moral goals, and finding a closer representation of the ideal equation.
That is valid, as long as both systems have the same goals. Marvin’s system includes the explicit goal “stay alive”, more heavily weighted then the goal “keep a stranger alive”; Fred’s system explicitly entirely excludes the goal “stay alive”.
If two moral systems agree both on the goals to be achieved, and the weightings to give those goals, then they will be the same moral system, yes. But two people’s moral systems need not agree on the underlying goals.
Again, I think I agree with Eliezer that a truly alien code of behavior, like that exhibited by sociopaths, and really inhuman aliens like the Pebblesorters or paperclippers, should maybe be referred to by some word other than morality. This is because since the word “morality” usually refers to doing things like making the world a happier place and increasing the positive things in life.
Well, to be fair, in a Paperclipper’s mind, paperclips are the positive things in life, and they certainly make the paperclipper happier. I realise that’s probably not what you intended, but the phrasing may need work.
Which really feeds into the question of what goals a moral system should have. To the Babyeaters, a moral system should have the goal of eating babies, and they can provide a lot of argument to support that point—in terms of improved evolutionary fitness, for example.
I think that we can agree that a moral system’s goals should be the good things in life. I’m less certain that we can agree on what those good things necessarily are, or on how they should be combined relative to each other. (I expect that if we really go to the point of thoroughly dissecting what we consider to be the good things in life, then we’ll agree more than we disagree; I expect we’ll be over 95% in agreement, but not quite 100%. This is what I generally expect for any stranger).
For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority.
Sorry, the only Cherryh I’ve read is “The Scapegoat.” I thought it gave a good impression of how alien values would look to humans, but wish it had given some more ideas about what it was that made elves think so differently.
I’d say sometimes A, and sometimes B. But I think that’s true even in the absence of mental disorders; I don’t think that the “ideal equation” necessarily sits somewhere hidden in the human psyche.
It’s not that I think there’s literally a math equation locked in the human psyche that encodes morality. It’s more like there are multiple (sometimes conflicting) moral values and methods for resolving conflicts between them and that the sum of these can be modeled as a large and complicated equation.
That is valid, as long as both systems have the same goals. Marvin’s system includes the explicit goal “stay alive”, more heavily weighted then the goal “keep a stranger alive”; Fred’s system explicitly entirely excludes the goal “stay alive”.
You gave me the impression that Marvin valued “staying alive” less as an end in itself, and more as a means to achieve the end of improving the world. in particular when you said this:
Marvin’s moral system considers the total benefit to the world of every action; but he tends to weight actions in favour of himself, because he knows that in the future, he will always choose to do the right thing (by his morality) and thus deserves ties broken in his favour.
This is actually something that bothers me in fiction when a character who is superhumanly good and power (i.e. Superman, the Doctor) risks their lives to save a relatively small amount of people. It seems short-sighted of them to do that since they regularly save much larger groups of people and anticipate continuing to do so in the future, so it seems like they should preserve their lives for those people’s sakes.
Well, to be fair, in a Paperclipper’s mind, paperclips are the positive things in life, and they certainly make the paperclipper happier.
I get the impression that the paperclipper doesn’t feel happiness, just a raw motivation to increase the amount of paperclips.
I think that we can agree that a moral system’s goals should be the good things in life. I’m less certain that we can agree on what those good things necessarily are, or on how they should be combined relative to each other.
If you define “the good things in life” as “whatever an entity wants the most,” then you can agree, whatever someone wants is “good,” be it paperclips or eudaemonia. On the other hand, I’m not sure we should do this, there are some hypothetical entities I can imagine where I can’t see it as ever being good that they get what they want. For instance I can imagine a Human-Torture-Maximizer that wants to do nothing but torture human beings. It seems to me that even if there were a trillion Human-Torture-Maximizers and one human in the universe it would be bad for them to get what they want.
For more neutral, but still alien preferences, I’m less sure. It seems to me that I have a right to stop Human-Torture-Maximizers from getting what they want. But would I have the right to stop paperclippers? Making the same paperclip over and over again seems like a pointless activity to me, but if the paperclippers are willing to share part of the universe with existing humans do I have a right to stop them? I don’t know, and I don’t think Eliezer does either.
(I expect that if we really go to the point of thoroughly dissecting what we consider to be the good things in life, then we’ll agree more than we disagree; I expect we’ll be over 95% in agreement, but not quite 100%. This is what I generally expect for any stranger).
I think that we, and most humans, have the same basic desires, where we differ is the object of those desires, and the priority of those desires.
For instance, most people desire romantic love. But those desires usually have different objects, I desire romantic love with my girlfriend, other people desire it with their significant others. Similarly, most people desire to consume stories, but the object of that desire differs, some people like Transformers, others The Notebook.
Similarly, people often desire the same things, but differ as to their priorities, how much of those things they want. Most people desire both socializing, and quiet solitude, but some extroverts want lots of one and less of the other, while introverts are the opposite.
In the case of the paerclippers, my first instinct is to regard opposing paperclipping as no different from the many ways humans have persecuted each other for wanting different things in the past. But then it occurred to me that paperclip-maximizing might be different because most persecutions in the past involve persecuting people who have different objects and priorities, not people who actually have different desires. For instance homosexuality is the same kind of desire as heterosexuality, just with a different object (same sex instead of opposite).
For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority.
This seems like a difference in priority, rather than desire, as most people would prefer differing proportions of both both. It’s still a legitimate disagreement, but I think it’s more about finding a compromise between conflicting priorities, rather than totally different values.
Compounding this problem is the fact that people value diversity to some extent. We don’t value all types of diversity obviously, I think we’d all like to live in a world where people held unanimous views on the unacceptability of torturing innocent people. But we would like other people to be different from us in some ways. Most people, I think, would rather live in a world full of different people with different personalities than a world consisting entirely of exact duplicates (in both personality and memory) of one person. So it might be impossible to reach full agreement on those other values without screwing up the achievement of the Value of Diversity.
It’s not that I think there’s literally a math equation locked in the human psyche that encodes morality. It’s more like there are multiple (sometimes conflicting) moral values and methods for resolving conflicts between them and that the sum of these can be modeled as a large and complicated equation.
I’m sorry, there’s an ambiguity there—when you say “the sum of these”, are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
You gave me the impression that Marvin valued “staying alive” less as an end in itself, and more as a means to achieve the end of improving the world. in particular when you said this:
You are quite correct. I apologise; I changed that example several times from where I started, and it seems that one of my last-minute changes actually made it a worse example (my aim was to try to show how the explicit aim of self-preservation could be a reasonable moral aim, but in the process I made it not a moral aim at all). I should watch out for that in the future.
This is actually something that bothers me in fiction when a character who is superhumanly good and power (i.e. Superman, the Doctor) risks their lives to save a relatively small amount of people. It seems short-sighted of them to do that since they regularly save much larger groups of people and anticipate continuing to do so in the future, so it seems like they should preserve their lives for those people’s sakes.
I’ve always felt that was because one of the effects of great power, is that it’s so very easy to let everyone die. With great power, as Spiderman is told, comes great responsibility; one way to ensure that you’re not letting your own power go to your head, is by refusing to not-rescue anyone. After all, if the average science hero lets everyone he thinks is an idiot die, then who would be left?
Sometimes there’s a different reason, though; Sherlock Holmes would ignore a straightforward and safe case to catch a serial killer in order to concentrate on a tricky and deadly case involving a stolen diamond; he wasn’t in the detective business to help people, he was in the detective business in order to be challenged, and he would regularly refuse to take cases that did not challenge him.
(That’s probably a fair example as well, actually; for Holmes, only the challenge, the mental stimulation of a worthy foe, is important; for Superman, what is important is the saving of lives, whether from a mindless tsunami or Lex Luthor’s latest plot).
I think that we, and most humans, have the same basic desires, where we differ is the object of those desires, and the priority of those desires.
Hmmm. If you’re willing to accept zero, or near-zero, as a priority, then that statement can apply to any two sets of desires. Consider Sherlock holmes and a paperclipper; Holmes’ desire for mental stimulation is high-priority, his desire for paperclips is zero-priority, while the paperclipper’s desire for paperclips is high priority, and its desire for mental stimulation is zero-priority. (Some desires may have negative priority, which can then be interpreted as a priority to avoid that outcome—for example, my desire to immerse my hand in acid is negative, but a masochist may have a positive priority for that desire)
This implies that, in order to meaningfully differentiate the above statement from “some people have different desires”, I may have to designate some very low priority, below which the desire is considered absent (I may, of course, place that line at exactly zero priority). Some desires, however, may have no priority on their own, but inherit priority from another desire that they feed into; for example, a paperclipper has zero desire for self-preservation on its own, but it will desire self-preservation so that it can better create more paperclips.
Now, given a pool of potential goals, most people will pick out several desires from that pool, and there will be a large overlap between any two people (for example, most humans desire to eat—most but not all, there are certain eating disorders that can mess with that), and it is possible to pick out a set of desires that most people will have high priorities for.
It’s even probably possible to pick out a (smaller) set of desires such that those who do not have those desires at some positive priority are considered psychologically unhealthy. But such people nonetheless do exist.
Does this mean it isn’t bad to oppose paperclipping? I don’t know, maybe, but maybe not.
In my personal view, it is neutral to paperclip or to oppose paperclipping. It becomes bad to paperclip only when the paperclipping takes resources away from something more important.
And there are circumstances (somewhat forced circumstances) where it could be good to paperclip.
For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority.
This seems like a difference in priority, rather than desire, as most people would prefer differing proportions of both both. It’s still a legitimate disagreement, but I think it’s more about finding a compromise between conflicting priorities, rather than totally different values.
There exist people who would place negative value on the idea of following the instuctions of any legitimate authority. (They tend to remain a small and marginal group, because they cannot in turn form an authority for followers to follow without rampant hypocrisy).
Compounding this problem is the fact that people value diversity to some extent. We don’t value all types of diversity obviously, I think we’d all like to live in a world where people held unanimous views on the unacceptability of torturing innocent people. But we would like other people to be different from us in some ways. Most people, I think, would rather live in a world full of different people with different personalities than a world consisting entirely of exact duplicates (in both personality and memory) of one person. So it might be impossible to reach full agreement on those other values without screwing up the achievement of the Value of Diversity.
Yes, diversity has many benefits. The second-biggest benefit of diversity is that some people will be more correct than others, and this can be seen in the results they get; then everyone can re-diversify around the most correct group (a slow process, taking generations, as the most successful group slowly outcompetes the rest and thus passes their memes to a greater and/or more powerful proportion of the next generation). By a similar tokem, it means that one something happens that detroys one type of person, it doesn’t destroy everyone (bananas have a definite problem there, being a bit of a monoculture).
The biggest benefit is that it leads to social interaction. A completely non-diverse society would have to be a hive mind (or different experiences would slowly begin to introduce diversity), and it would be a very lonely hive mind, with no-one to talk to.
I’m sorry, there’s an ambiguity there—when you say “the sum of these”, are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
Nearly all of humanity as a whole. There are obviously some humans who don’t really value morality, we call them sociopaths, but I think most humans care about very similar moral concepts. The fact that people have somewhat different personal preferences and desires at first might seem to challenge this idea, but I don’t really think it does. It just means that there are some desires that generate the same “value” of “good” when fed into the “equation.” In fact, if diversity is a good, as we discussed previously, then people having different personal preferences might in fact be morally desirable.
Hmmm. If you’re willing to accept zero, or near-zero, as a priority, then that statement can apply to any two sets of desires......This implies that, in order to meaningfully differentiate the above statement from “some people have different desires”, I may have to designate some very low priority, below which the desire is considered absent
That’s a good point. I was considering using the word “proportionality” instead of “priority” to better delineate that I don’t accept zero as a priority, but rejected it because it sounded clunky. Maybe I shouldn’t have.
In my personal view, it is neutral to paperclip or to oppose paperclipping. It becomes bad to paperclip only when the paperclipping takes resources away from something more important.
I agree with that. What I’m wondering is, would I have a moral duty to share resources with a paperclipper if it existed, or would pretty much any of the things I spend the resources on if I kept them for myself count (i.e. eudaemonic things) as “something more important.”
There exist people who would place negative value on the idea of following the instuctions of any legitimate authority.
I think there might actually be lots of people like this, but most appear normal because they place even greater negative value on doing something stupid because they ignored good advice just because it came from an authority. In other words, following authority is a negative terminal value, but an extremely positive instrumental value.
The biggest benefit is that it leads to social interaction. A completely non-diverse society would have to be a hive mind (or different experiences would slowly begin to introduce diversity), and it would be a very lonely hive mind, with no-one to talk to.
Exactly. I would still want the world to be full of a diverse variety of people, even if I had a nonsentient AI that was right about everything and could serve my every bodily need.
I’m sorry, there’s an ambiguity there—when you say “the sum of these”, are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
Nearly all of humanity as a whole. There are obviously some humans who don’t really value morality, we call them sociopaths, but I think most humans care about very similar moral concepts.
Okay then, next question; how do you decide which people to exclude? You say that you are excluding sociopaths, and I think that they should be excluded; but on exactly what basis? If you’re excluding them simply because they fail to have the same moral imperatives as the ones that you think are important, then that sounds very much like a No True Scotsman argument to me. (I exclude them mainly on an argument of appeal to authority, myself, but that also has logic problems; in either case, it’s a matter of first sketching out what the moral imperative should be, then throwing out the people who don’t match).
And for a follow-up question; is it necessary to limit it to humanity? Let us assume that, ten years from now, a flying saucer lands in the middle of Durban, and we meet a sentient alien form of life. Would it be necessary to include their moral preferences in the equation as well?
Even if they are Pebblesorters?
In fact, if diversity is a good, as we discussed previously, then people having different personal preferences might in fact be morally desirable.
It may be, but only within a limited range. A serial killer is well outside that range, even if he believes that he is doing good by only killing “evil” people (for some definition of “evil”).
What I’m wondering is, would I have a moral duty to share resources with a paperclipper if it existed, or would pretty much any of the things I spend the resources on if I kept them for myself count (i.e. eudaemonic things) as “something more important.”
Hmmm. I think I’d put “buying a packet of paperclips for the paperclipper” as on the same moral footing, more or less, as “buying an icecream for a small child”. It’s nice for the person (or paperclipper) recieving the gift, and that makes it a minor moral positive by increasing happiness by a tiny fraction. But if you could otherwise spend that money on something that would save a life, then that clearly takes priority.
I think there might actually be lots of people like this, but most appear normal because they place even greater negative value on doing something stupid because they ignored good advice just because it came from an authority. In other words, following authority is a negative terminal value, but an extremely positive instrumental value.
Hmmm. Good point; that is quite possible. (Given how many people seem to follow any reasonably persuasive authority, though, I suspect that most people have a positive priority for this goal—this is probably because, for a lot of human history, peasants who disagreed with the aristocracy tended to have fewer descendants unless they all disagreed and wiped out said aristocracy).
Exactly. I would still want the world to be full of a diverse variety of people, even if I had a nonsentient AI that was right about everything and could serve my every bodily need.
Here’s a tricky question—what exactly are the limits of “nonsentient”? Can a nonsentient AI fake it by, with clever use of holograms and/or humanoid robots, cause you to think that you are surrounded by a diverse variety of people even when you are not (thus supplying the non-bodily need of social interaction)? The robots would all be philosophical zombies, of course; but is there any way to tell?
Okay then, next question; how do you decide which people to exclude?
I don’t think I’m coming across right. I’m not saying that morality is some sort of collective agreement of people in regards to their various preferences. I’m saying that morality is a series of concepts such as fairness, happiness, freedom etc., that these concepts are objective in the sense that it can be objectively determined how much fairness, freedom, happiness etc. there is in the world, and that the sum of these concepts can be expressed as a large equation.
People vary in their preference for morality, most people care about fairness, freedom, happiness, etc. to some extent. But there are some people who don’t care about morality at all, such as sociopaths.
Morality isn’t a preference. It isn’t the part of a person’s brain that says “This society is fair and free and happy, therefore I prefer it.” Morality is those disembodied concepts of freedom, fairness, happiness, etc. So if a person doesn’t care about those things, it doesn’t mean that freedom, fairness, happiness, etc. aren’t part of their morality. It means that person doesn’t care about morality, they care about something else.”
To use the Pebblesorter analogy again, the fact that you and I don’t care about sorting pebbles into prime-numbered heaps isn’t because we have our own concept of “primeness” that doesn’t include 2, 3, 5 and 7. It just means we don’t care about primeness.
To make another analogy, if most people preferred wearing wool clothes but one person preferred cotton, that wouldn’t mean that that person had their own version of wool, which was cotton. It means that that person doesn’t prefer wool.
Look inward, and consider why you think most people should be included. Presumably it’s because you really care a lot about being fair. But that necessarily means that you cared about fairness before you even considered what other people might think. Otherwise it wouldn’t have even occurred to you to think about what they preferred in the first place.
The fact that most humans care, to some extent, about the various facets of morality, is a very lucky thing, a planet full of sociopaths would be most unpleasant. But it isn’t relevant to the truth of morality. You’d still think torturing people was bad if all the non-sociopaths on Earth except you were killed, wouldn’t you? If, in that devastated world, you came across a sociopath torturing another sociopath or an animal, and could stop them at no risk to yourself, you’d do it, wouldn’t you?
You say that you are excluding sociopaths, and I think that they should be excluded; but on exactly what basis?
I suspect that your intuition comes from the fact that a central part of morality is fairness, and sociopaths don’t care about fairness. Obviously being fair to the unfair is as unwise as tolerating the intolerant.
And for a follow-up question; is it necessary to limit it to humanity? Let us assume that, ten years from now, a flying saucer lands in the middle of Durban, and we meet a sentient alien form of life. Would it be necessary to include their moral preferences in the equation as well?
Again, I want to emphasize that morality isn’t the “preference” part, it’s the “concepts” part. But the question of the moral significance of aliens is relevant, I think it would depend on how many of the concepts that make up morality they cared about. I think that at a bare minimum they’d need fairness and sympathy.
So if the Pebblesorters that came out of that ship were horrified that we didn’t care about primality, but were willing to be fair and share the universe with us, they’d be a morally worthwhile species. But if they had no preference for fairness or any sympathy at all, and would gladly kill a billion humans to sort a few more pebbles, that would be a different story. In that case we should probably, after satisfying ourselves that all Pebblesorters were psychologically similar, start prepping a Relativistic Kill Vehicle to point at their planet if they try something.
Here’s a tricky question—what exactly are the limits of “nonsentient”? Can a nonsentient AI fake it by, with clever use of holograms and/or humanoid robots, cause you to think that you are surrounded by a diverse variety of people even when you are not (thus supplying the non-bodily need of social interaction)? The robots would all be philosophical zombies, of course; but is there any way to tell?
I don’t know if I could tell, but I’d very much prefer that the AI not do that, and would consider myself to have been massively harmed if it did, even if I never found out. My preference is to actually interact with a diverse variety of people, not to merely have a series of experiences that seem like I’m doing it.
So, OK. Suppose, on this account, that you and I both care about morality to the same degree… that is, you don’t care about morality more than I do, and I don’t care about morality more than you do. (I’m not sure how we could ever know that this was the case, but just suppose hypothetically that it’s true.)
Suppose we’re faced with a situation in which there are two choices we can make. Choice A causes a system to be more fair, but less free. Choice B leaves that system unchanged. Suppose, for simplicity’s sake, that those are the only two choices available, and we both have all relevant information about the system.
On your account, will we necessarily agree on which choice to make? Or is it possible, in that situation, that you might choose A and I choose B, or vice-versa?
I think it depends on the degree of the change. If the change is very lopsided (i.e −100 freedom, +1 fairness) I think we’d both choose B.
If we assume that the degree of change is about the same (i.e. +1 fairness, −1 freedom) it would depend on how much freedom and fairness already exist. If the system is very fair, but very unfree, we’d both choose B, but if it’s very free and very unfair we’d both choose A.
However, if we are to assume that the gain in fairness and the loss in freedom are of approximately equivalent size and the current system has fairly large amounts of both freedom and fairness (which I think is what you meant) then it might be possible that we’d have a disagreement that couldn’t be resolved with pure reasoning.
This is called moral pluralism, the idea that there might be multiple moral values (such as freedom, fairness, and happiness) which are objectively correct, imperfectly commensurable with each other, and can be combined in different proportions that are of approximately equivalent objective moral value. If this is the case then your preference for one set of proportions over the other might be determined by arbitrary factors of your personality.
This is not the same as moral relativism, as these moral values are all objectively good, and any society that severely lacks one of them is objectively bad. It’s just that there are certain combinations with different proportions of values that might be both “equally good,” and personal preferences might be the “tiebreaker.” To put it in more concrete terms, a social democracy with low economic regulation and a small welfare state might be “just as good” as a social democracy with slightly higher economic regulation and a slightly larger welfare state, and people might honestly and irresolvably disagree over which one is better. However, both of those societies would definitely be objectively better than Cambodia under the Khmer Rouge, and any rational, fully informed person who cares about morality would be able to see that.
Of course, if we are both highly rational and moral, and disagreed about A vs. B, we’d both agree that fighting over them excessively would be morally worse than choosing either of them, and find some way to resolve our disagreement, even if it meant flipping a coin.
I agree with you that in sufficiently extreme cases, we would both make the same choice. Call that set of cases S1.
I think you’re saying that if the case is not that extreme, we might not make the same choice, even though we both care equally about the thing you’re using “morality” to refer to. I agree with that as well. Call that set of cases S2.
I also agree that even in S2, there’s a vast class of options that we’d both agree are worse than either of our choices (as you illustrate with the Khmer Rouge), and a vast class of options that we’d both agree are better than either of our choices, supposing that we are as you suggest rational informed people who care about the thing you’re using “morality” to refer to.
If I’m understanding you, you’re saying in S2 we are making different decisions, but our decisions are equally good. Further, you’re saying that we might not know that our decisions are equally good. I might make choice A and think choice B is wrong, and you might make choice B and think choice A is wrong. Being rational and well-informed people we’d agree that both A and B are better than the Khmer Rouge, and we might even agree that they’re both better than fighting over which one to adopt, but it might still remain true that I think B is wrong and you think A is wrong, even though neither of us thinks the other choice is as wrong as the Khmer Rouge, or fighting about it, or setting fire to the building, or various other wrong things we might choose to evaluate.
It follows that if a choice can go one of three ways (c1, c2, c3) and if I think c1> c2 > c3 and therefore endorse c1, and if you think c2 > c1 > c3 and therefore endorse c2, and if we’re both rational informed people who are in possession of the same set of facts about that choice and its consequences, and if we each think that the other is wrong to endorse the choice we endorse (while still agreeing that it’s better than c3), that there are (at least) two possibilities.
One possibility is that c1 and c2 are, objectively, equally good choices, but we each think the other is wrong anyway. In this case we both care about morality, even though we disagree about right action.
Another possibility is that c1 and c2 are, objectively, not equally good. For example, perhaps c1 is objectively bad, violates morality, and I endorse it only because I don’t actually care about morality. Of course, in this case I may use the label “morality” to describe what I care about, but that’s at best confusing and at worst actively deceptive, because what I really care about isn’t morality at all, but some other thing, like prime-numbered heaps or whatever.
Yes?
So, given that, I think my question is: how might I go about figuring out which possibility is the case?
One possibility is that c1 and c2 are, objectively, equally good choices, but we each think the other is wrong anyway.
I’d say it’s misleading to say we thought the other person was “wrong,” since in this context “wrong” is a word usually used to describe a situation where someone is in objective moral error. It might be better to say: “c1 and c2 are, objectively, equally morally good, but we each prefer a different one for arbitrary, non-moral reasons.”
This doesn’t change your argument in any way, I just think it’s good to have the language clear to avoid accidentally letting in any connotations that don’t belong.
So, given that, I think my question is: how might I go about figuring out which possibility is the case?
This is not something I have done a lot of thinking on since the odds of ever encountering such a situation are quite low at the present. It seems to me, however, that if you are this fair to your opponent, and care this much about finding out the honest truth, that you probably care at least somewhat about morality.
(This brings up an interesting question, which is: might there be some “semi-sociopathic” humans who care about morality, but incrementally, not categorically? That is, if one of these people was rational, fully informed, lacking in self deception, and lacked akrasia, they would devote maybe 70% of their time and effort to morality and 30% to other things? Such a person, if compelled to be honest, might admit that c2 is morally worse than c1, but they don’t care because they’ve used up their 70% of moral effort for the day. It doesn’t seem totally implausible that such people might exist, but maybe I’m missing something about how moral psychology works, maybe it doesn’t work unless it’s all or nothing.)
As for determining if your opponent care about morality, you might look to see if they exhibit any of the signs of sociopathy. You might search their arguments for signs of anti-epistemology, or plain moral errors. If you don’t notice any of these things, you might assign a higher probability to prediction that your disagreement is to preferring different forms of pluralism
Of course, in real life perfectly informed, rational humans who lack self deception, akrasia, and so on do not exist. So you should probably assign a much, much, much higher probability to one of those things causing your disagreement.
It might be better to say: “c1 and c2 are, objectively, equally morally good, but we each prefer a different one for arbitrary, non-moral reasons.”
OK. In which case I can also phrase my question as, when I choose c1 over c2, how can I tell whether I’m making that choice for objective moral reasons, as opposed to making that choice for arbitrary non-moral reasons?
You’re right that it doesn’t really change the argument, I’m just trying to establish some common language so we can communicate clearly.
For my own part, I agree with you that ignorance and akrasia are major influences, and I also believe that what you describe as “incremental caring about morality” is pretty common (though I would describe it as individual values differing).
Wikipedia’s page on internalism and externalism calls an entity that understands moral arguments, but is not motivated by them an “amoralist.” We could say that a person who cares about morality incrementally to have individual values that are part moralist and part amoralist.
It’s hard to tell how many people are like this due to the confounding factors of irrationality and akrasia. But I think it’s possible that there are some people who, if their irrationality and akrasia were cured, would not act perfectly moral. These people would say “I know that the world would be a better place if I acted differently, but I only care about the world to a limited extent.”
However, considering that these people would be rational and lack akrasia, they would still probably do more moral good than the average person does today.
These people would say “I know that the world would be a better place if I acted differently, but I only care about the world to a limited extent.”
Would they say necessarily say that? Or might they instead say “I know you think the world would be a better place if I acted differently, but actually it seems to me the world is better if I do what I’m doing?”
Would they say necessarily say that? Or might they instead say “I know you think the world would be a better place if I acted differently, but actually it seems to me the world is better if I do what I’m doing?”
That depends on if they are using the term “better” to mean “morally better” or “more effectively satisfies the sum total of all my values, both moral or non-moral.”
If the person is fully rational, lacking in self-deception, is being totally honest with you, and you and they had both agreed ahead of time that the word “better” means “morally better” then yes, I think they would say that. If they were lying, or they thought you were using the term “better” to mean “more effectively satisfies the sum total of all my values, both moral or non-moral,” then they might not.
I agree with you that IF (some of) my values are not moral and I choose to maximally implement my values, THEN I’m choosing not to act so as to make the world a morally better place, and IF I somehow knew that those values were not moral, THEN I would say as much if asked (supposing I was aware of my values and I was honest and so forth).
But on your account I still don’t see any way for me to ever know which of my values are moral and which ones aren’t, no matter how self-aware, rational, or lacking in self-deception I might be.
Also, even if I did somehow know that, and were honest and so forth, I don’t think I would say “I only care about the world to a limited extent.” By maximizing my values as implemented in the world, I would be increasing the value of the world, which is one way to express caring about the world. Rather, I would say “I only care about moral betterness to a limited extent; there are more valuable things for the world to be than morally better.”
But on your account I still don’t see any way for me to ever know which of my values are moral and which ones aren’t, no matter how self-aware, rational, or lacking in self-deception I might be.
How does a Pebblesorter know it’s piles are prime? The less intelligent and rational probably use some sort of vague intuition. The more intelligent and rational probably try dividing the numbers of pebbles by a number other than one.
If you had full knowledge of the concept of “morality” and all the various sub-concepts it included, you could translate that concept into a mathematical equation (the one I’ve been discussing with CC lately), and see if the various values of yours that you feed into it return positive numbers.
If your knowledge is more crude (i.e. if you’re a real person who actually exists), then a possible way to do it would be to divide the nebulous super-concept of “morality” into a series of more concrete and clearly defined sub-concepts that compose it (i.e. freedom, happiness, fairness, etc.). It might also be helpful to make a list of sub-concepts that are definitely not part of morality (possible candidates include malice, sadism, anhedonia, and xenophobia).
After doing that you could, if you are not self-deceived, use introspection to figure what you value. If you find that your values include the various moral sub-concepts, then it seems like you value morality. If you find yourself not valuing the moral sub-concepts, or valuing some nonmoral concept, then you do not value morality, or value non-moral things.
There is no pure ghostly essence of goodness apart from things like truth, happiness and sentient life.
The moral equation we are looking for isn’t something that will provide us with a ghostly essence. It is something that will allow us to sum up and aggregate all the seperate good things like truth, happiness, and sentient life, so that we can effectively determine how good various combinations of these things are relative to each other, and reach an optimal combo.
Do you want people to be happy, free, be treated fairly, etc? Then you value morality to some extent. Do you love torturing people just for the hell of it, or want to convert all the matter in the universe into paperclips? Then you, at the very least, definitely value other things than morality.
Also, even if I did somehow know that, and were honest and so forth, I don’t think I would say “I only care about the world to a limited extent.” By maximizing my values as implemented in the world, I would be increasing the value of the world, which is one way to express caring about the world.
By “caring” I meant “caring about whether the world is a good and moral place.” If you instead use the word “caring” to mean “have values that assign different levels of desirability to various possible states that the world could be in” then you are indeed correct that you would not say you didn’t care about the world.
Rather, I would say “I only care about moral betterness to a limited extent; there are more valuable things for the world to be than morally better.”
If by “valuable” you mean “has more of the things that I care about,” then yes, you could say that. Remember, however, that in that case what is “valuable” is subjective, it changes from person to person depending on their individual utility functions. What is “morally valuable,” by contrast, is objective. Anyone regardless of their utility function, can agree on whether or not the world has great quantities of things like truth, freedom, happiness, and sentient life. What determines the moral character of a person is how much they value those particular things.
Also, as an aside, when I mentioned concepts that probably aren’t part of morality earlier, I did not mean to say that pursuit of those concepts always neccessarily leads to immoral results. For instance, imagine a malicious sadist who wants to break someone’s knees. This person assaults someone else out of pure malice and breaks their knees. The injured person turns out to be an escaped serial killer who was about to kill again, and the police are able to apprehend them in their injured state. In this case the malicious person has done good. However, this is not because they have intentionally increased the amount of malicious torture in the universe. It is because they accidentally decreased the amount of murders in the universe.
I 100% agree that there is no ghostly essence of goodness.
I agree that pursuing amoral, or even immoral, values can still lead to moral results. (And also vice-versa.)
I agree that if I somehow knew what was moral and what wasn’t, then I would have a basis for formally distinguishing my moral values from my non-moral values even when my intuitions failed. I could even, in principle, build an automated mechanism for judging things as moral or non-moral. (Similarly, if a Pebblesorter knew that primeness was what it valued and knew how to factor large numbers, it would have a basis for formally distinguishing piles it valued from piles it didn’t value even when its intuitive judgments failed, and it could build an automated mechanism for distinguishing such piles.)
I agree with you that real people who actually exist can’t do this, at least not in detail.
You suggest we can divide morality into subconcepts that comprise it (freedom, happiness, fairness, etc.) and that it excludes (anhedonia, etc.). What I continue to not get is, on your account, how I do that in such a way as to ensure that what I end up with is the objectively correct list of moral values, which on your account exists, rather than some different list of values.
That is, suppose Sam and George both go through this exercise, and one of them ends up with “freedom” on their list but not “cooperation”, and the other ends up with “cooperation” but not “freedom.” On your account it seems clear that at least one of them is wrong, because the correct list of moral values is objective.
So, OK… what would we expect to experience if Sam were right? How does that differ from what we would expect to experience if George were right, or if neither of them were?
Do you want people to be happy, free, be treated fairly, etc? Then you value morality to some extent.
Again: how do we know that? What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does? What should I be looking for, to notice if this is true, or confirm that it isn’t? I would still want people to be happy, free, be treated fairly, etc. in either case, after all. What differences would I experience between the two cases?
If you instead use the word “caring” to mean “have values that assign different levels of desirability to various possible states that the world could be in”
Yes, that’s more or less what I mean by “caring”. More precisely I would say that caring about X consists of desiring states of the world with more X more than states of the world with less X, all else being equal, but that’s close enough to what you said.
If by “valuable” you mean “has more of the things that I care about,” then yes, you could say that. Remember, however, that in that case what is “valuable” is subjective, it changes from person to person depending on their individual utility functions.
Yes, that’s what I mean by “valuable.” And yes, absolutely, what is valuable changes from person to person. If I act to maximize my values and you act to maximize yours we might act in opposition (or we might not, depending, but it’s possible).
And I get that you want to say that if we both gave up maximizing our values and instead agreed to implement moral values, then we would be cooperating instead, and the world would be better (even if it turned out that both of us found it less valuable). What I’m asking you is how (even in principle) we could ever reach that point.
To say that a little differently: you value some things (Vg) and I value some things (Vd). Supposing we are both perfectly rational and honest and etc., we can both know what Vg and Vd are, and what events in the world would maximize each. We can agree to cooperate on maximizing the intersection of (Vg,Vd), and we can work out some pragmatic compromise about the non-overlapping stuff. So far so good; I see how we could in principle reach that point, even if in practice we aren’t rational or self-aware or honest enough to do it.
But I don’t see how we could ever say “There’s this other list, Vm, of moral values; let’s ignore Vg and Vd altogether and instead implement Vm!” because I don’t see how we could ever know what Vm was, even in principle. If we happened to agree on some list Vm, either by coincidence or due to social conditioning or for other reasons, we could agree to implement Vm… which might or might not make the world better, depending on whether Vm happened to be the objectively correct list of moral values. But I don’t see how we could ever, even in principle, confirm or deny this, or correct it if we somehow came to know we had the objectively wrong list.
And if we can’t know or confirm or deny or correct it, even in principle, then I don’t see what is added by discussing it. It seems to me I can just as usefully say, in this case, “I value happiness, freedom, fairness, etc. I will act to maximize those values, and I endorse acting this way,” and nothing is added by saying “Those values comprise morality” except that I’ve asserted a privileged social status for my values.
So, OK… what would we expect to experience if Sam were right? How does that differ from what we would expect to experience if George were right, or if neither of them were?......Again: how do we know that? What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does?
Well, I am basically asserting that morality is some sort of objective equation, or “abstract idealized dynamic,” as Eliezer calls it, concerned with people’s wellbeing. And I am further asserting that most human beings care very much about this concept. I think this would make the following predictions:
In a situation where a given group of humans had similar levels of empirical knowledge and a similar sanity waterline there would be far more moral agreement among them than would be predicted by chance, and far less moral disagreement than is mentally possible.
It is physically possible to persuade people to change their moral values by reasoned argument.
Inhabitants of a society who are unusually rational and intelligent will be the first people in that society to make moral progress, as they will be better at extrapolating answers out of the “equation.”
If one attempted to convert the moral computations people make into an abstract, idealized process, and determine it’s results, many people would find those results at least somewhat persuasive, and may find their ethical views changed by observing them.
All of these predictions appear to be true:
Human societies tend to have a rather high level of moral agreement between their members. Conformity is not necessarily an indication of rightness, it seems fairly obvious that whole societies have held gravely mistaken moral views, such as those that believed slavery was good. However, it is interesting that all those people in those societies were mistaken in exactly the same way. That seems like evidence that they were all reasoning towards similar conclusions, and the mistakes they made were caused by common environmental factors that impacted all of them. There are other theories that explain this data of course, (peer pressure, for instance), but I still find it striking.
I’ve had moral arguments made by other people change my mind, and changed the minds of other people by moral argument. I’m sure you have also had this experience.
It is well known that intellectuals tend to develop and adopt new moral theories before the general populace does. Common examples of intellectuals whose moral concepts have disseminated into the general populace include John Locke, Jeremy Bentham, and William Lloyd Garrison. Many of these peoples’ principles have since been adopted into the public consciousness.
Ethical theorists who have attempted to derive new ethical principles by working from an abstract, idealized form of ethics have often been very persuasive. To name just one example, Peter Singer ended up turning thousands of people into vegetarians with moral arguments that started on a fairly abstract level.
It seems to me I can just as usefully say, in this case, “I value happiness, freedom, fairness, etc. I will act to maximize those values, and I endorse acting this way,” and nothing is added by saying “Those values comprise morality”
Asserting that those values comprise morality seems to be effective because it seems to most people that those values are related in some way, because they form the superconcept “morality.” Morality is a useful catchall term for certain types of values, and it would be a shame to lose it.
Still, I suppose that asserting “I value happiness, freedom, fairness, etc” is similar enough to saying “I care about morality” that I really can’t object terribly strongly if that’s what you’d prefer to do.
except that I’ve asserted a privileged social status for my values.
Why does doing that bother you? Presumably, because you care about the moral concept of fairness, and don’t want to claim an unfair level of status for you and your views. But does it really make sense to say “I care about fairness, but I want to be fair to other people who don’t care about it, so I’ll go ahead and let them treat people unfairly, in order to be fair.” That sounds silly, doesn’t it? it has the same problems that come with being tolerant of intolerant people.
I think this would make the following predictions:
All of those predictions seem equally likely to me whether Sam is right or George is, so don’t really engage with my question at all. At this point, after several trips ’round the mulberry bush, I conclude that this is not because I’m being unclear with my question but rather because you’re choosing not to answer it, so I will stop trying to clarify the question further.
If I map your predictions and observations to the closest analogues that make any sense to me at all, I basically agree with them.
I suppose that asserting “I value happiness, freedom, fairness, etc” is similar enough to saying “I care about morality” that I really can’t object terribly strongly if that’s what you’d prefer to do.
It is.
Why does doing that [asserting a privileged social status for my values] bother you?
It doesn’t bother me; it’s a fine thing to do under some circumstances. If we can agree that that’s what we’re doing when we talk about “objective morality,” great. If not (which I find more likely), never mind.
Presumably, because you care about the moral concept of fairness, and don’t want to claim an unfair level of status for you and your views.
As above, I don’t see what the word “moral” is adding to this sentence. But sure, unfairly claiming status bothers me to the extent that I care about fairness. (That said, I don’t think claiming status by describing my values as “moral” is unfair; pretty much everybody has an equal ability to do it, and indeed they do. I just think it confuses any honest attempt at understanding what’s really going on when we decide on what to do.)
But does it really make sense to say “I care about fairness, but I want to be fair to other people who don’t care about it, so I’ll go ahead and let them treat people unfairly, in order to be fair.”
It depends on why and how I value (“care about”) fairness.
If I value it instrumentally (which I do), then it makes perfect sense to say that being fair to people who treat others unfairly is net-valuable, although it might be true or false in any given situation depending on what is achieved by the various kind of fairness that exist in tension in that situation.
Similarly, if I value it in proportion to how much of it there is (which I do), then it makes sense to say that, although it might be true or false depending on how much fairness is gained or lost by doing so.
That sounds silly, doesn’t it?
(nods) Totally. And the ability to phrase ideas in silly-sounding ways is valuable for rhetorical purposes, although it isn’t worth much as an analytical tool.
All of those predictions seem equally likely to me whether Sam is right or George is, so don’t really engage with my question at all.
I’m really sorry, I was trying to kill two birds with one stone and simultaneously engage that question and your later question [“What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does?”] at the same time, and I ended up doing a crappy job of answering both of them. I’ll try to just answer the Sam and George question now.
I’ll start by examining the Pebblesorters P-George and P-Sam. P-George thinks 9 is p-right and 16 is p-wrong. P-Sam thinks 9 is p-wrong and 16 is p-right. They both think they are using the word “p-right” to refer to the same abstract, idealized process. What can they do to see which one is right?
They assume that most other Pebblesorters care about the same abstract process they do, so they can try to persuade them and see how successful they are. Of course, even if all the Pebblesorters agree with one of them, that doesn’t necessarily mean that one is p-correct, those sorters may be making the same mistake as P-George or P-Sam. But I think it’s non-zero Bayesian evidence of the p-rightness of their views.
They can try to control for environmentally caused error by seeing if they can also persuade Pebblesorters who live in different environments and cultures.
They can find the most rational and p-sane Pebblesorting societies and see if they have an easier time persuading them.
They can actually try to extrapolate what the abstract, idealized equation that the word “p-right” represents is and compare it to their views. They read up on Pebblesorter philospher’s theories of p-rightness and see how they correlate with their views. Pebblesorting is much simpler than morality, so we know that the abstract, idealized dynamic that the concept “p-right” represents is “primality.” So we know that P-Sam and P-George are both partly right and partly wrong, 9 and 16 both aren’t prime.
Now let’s translate that into human.
We would expect if Sam was right and George was wrong:
He would have an easier time persuading non-sociopathic humans of the rightness of his views than George, because his views are closer to the results of the equation those people have in their head.
If he went around to different societies with different moral views and attempted to persuade the people there of his views he should, on average, also have an easier time of it than George, again because his views are closer to the results of the equation those people have in their head.
Societies with higher levels of sanity and rationality should be especially easily persuaded, because they are better at determining what the results of that equation would be.
When Sam compared his and George’s views to views generated by various attempts by philosophers to create an abstract idealized version of the equation (ie. moral theories), his view should be a better match to many of them, and the results they generate, than George’s are.
The problem is that the concept of morality is far more complex than the concept of primality, so finding the right abstract idealized equation is harder for humans than it is for Pebblesorters. We still haven’t managed to do it yet. But I think that comparing Sam and George’s views to the best approximations we have so far (various forms of consequentialism, in my view) we can get some Bayesian evidence of the rightness of their views.
If George is right, he will achieve these results instead of Sam. If they are both wrong, they will both fail at doing these things.
If I value it instrumentally (which I do), then it makes perfect sense to say that being fair to people who treat others unfairly is net-valuable, although it might be true or false in any given situation depending on what is achieved by the various kind of fairness that exist in tension in that situation.
Sorry, I was probably being unclear as to what I meant because I was trying to sound clever. When I said it was silly to be fair to unfair people what I meant was that you should not regard their advice on how to best treat other people with the same consideration you’d give to a fair-minded person’s advice.
For instance, you wouldn’t say “I think it’s wrong to enslave black people, but that guy over there thinks it’s right, so let’s compromise and believe it’s okay to enslave them 50% of the time.” I suppose you might pretend to believe that if the other guy had a gun and you didn’t, but you wouldn’t let his beliefs affect yours.
I did not mean that, for example, if you, two fair-minded people, and one unfair-minded person are lost in the woods and find a pie, that you shouldn’t give the unfair-minded person a quarter of the pie to eat. That is an instance where it does make sense to treat unfair people fairly.
OK. Thanks for engaging with the question; that was very helpful. I now have a much better understanding of what you believe the differences-in-practice between moral and non-moral values are.
Just to echo back what I’m hearing you say: to the extent that some set of values Vm is easier to convince humans to adopt than other sets of values and easier to convince sane, rational societies to adopt than less sane, less rational societies and better approximates the moral theories created by philosophers than other sets of values, to that extent we can be confident that Vm is the set of values that comprise morality.
Did I get that right?
Regarding fairmindedness: I endorse giving someone’s advice consideration to the extent that I’m confident that considering their advice will implement my values. And, sure, it’s unlikely that the advice of an unfairminded person would, if considered, implement the value of fairness.
This brings up an interesting question, which is: might there be some “semi-sociopathic” humans who care about morality, but incrementally, not categorically?
It seems very likely that a person who cares a certain amount about morality, and a certain amount about money, would be willing to compromise his morality if given sufficient money. Such a mental model would form the basis of bribery. (It doesn’t have to be money, either, but the principle remains the same).
So a semi-sociopathic person would be anyone who could be bribed into completely disregarding morality.
a semi-sociopathic person would be anyone who could be bribed into completely disregarding morality.
On this account, we could presumably also categorize a semi-semi-sociopathic person as one who could be bribed into partially disregarding the thing we’re labeling “morality”. And of course bribes needn’t be money… people can be bribed by all kinds of things. Social status. Sex. Pleasant experiences. The promise of any or all of those things in the future.
Which is to say, we could categorize a semi-semi-sociopath as someone who cares about some stuff, and makes choices consistent with maximizing the stuff they care about, where some of that stuff is what we’re labeling “morality” and some of it isn’t.
We could also replace the term “semi-semi-sociopath” with the easier to pronounce and roughly equivalent term “person”.
It’s also worth noting that there probably exists stuff that we would label “morality” in one context and “bribe” in another, were we inclined to use such labels.
I don’t think I’m coming across right. I’m not saying that morality is some sort of collective agreement of people in regards to their various preferences. I’m saying that morality is a series of concepts such as fairness, happiness, freedom etc., that these concepts are objective in the sense that it can be objectively determined how much fairness, freedom, happiness etc. there is in the world, and that the sum of these concepts can be expressed as a large equation.
Ah, I think I see your point. What you’re saying—and correct me if I’m wrong—is that there is some objective True Morality, some complex equation that, if applied to any possible situation, will tell you how moral a given act is.
This is probably true.
This equation isn’t written into the human psyche; it exists independantly of what people think about morality. It just is. And even if we don’t know exactly what the equation is, even if we can’t work out the morality of a given act down to the tenth decimal place, we can still apply basic heuristics and arrive at a usable estimate in most situations.
My question is, then—assuming the above is true, how do we find that equation? Does there exist some objective method whereby you, I, a Pebblesorter, and a Paperclipper can all independently arrive at the same definition for what is moral (given that the Pebblesorter and Paperclipper will almost certainly promptly ignore the result)?
(I had thought that you were proposing that we find that equation by summing across the moral values and imperatives of humanity as a whole—excluding the psychopaths. This is why I asked about the exclusion, because it sounded a lot like writing down what you wanted at the end of the page and then going back and discarding the steps that wouldn’t lead there; that is also why I asked about the aliens).
I don’t know if I could tell, but I’d very much prefer that the AI not do that, and would consider myself to have been massively harmed if it did, even if I never found out. My preference is to actually interact with a diverse variety of people, not to merely have a series of experiences that seem like I’m doing it.
Yes, I think we’re in agreement on that. (Though this does suggest that ‘sentient’ may need a proper definition at some point).
What you’re saying—and correct me if I’m wrong—is that there is some objective True Morality, some complex equation that, if applied to any possible situation, will tell you how moral a given act is.
In the same way as there exists a True Set of Prime Numbers, and True Measure of How Many Paperclips There Are...
My question is, then—assuming the above is true, how do we find that equation?
Even though the equation exists independently of our thoughts (the same way primality exists independently from Pebblesorter thoughts) fact that we are capable of caring about the results given by the equation means we must have some parts of it “written” in our heads, the same way Pebblesorters must have some concept of primality “written” in their heads. Otherwise, how would we be capable of caring about its results?
I think that probably evolution metaphorically “wrote” a desire to care about the equation in our heads because if humans care about what is good and right it makes it easier for them to cooperate and trust each other, which has obvious fitness advantages. Of course, the fact that evolution did a good thing by causing us to care about morality doesn’t mean that evolution is always good, or that evolutionary fitness is a moral justification for anything. Evolution is an amoral force causes many horrible things to happen. It just happened that in this particular instance, evolution’s amoral metaphorical “desires” happened to coincide with what was morally good. That coincidence is far from the norm, in fact, evolution probably deleted morality from the brains of sociopaths because double-crossing morally good people also sometimes confers a fitness advantage.
So how do we learn more about this moral equation that we care about? One common form of attempting to get approximations of it in philosophy is called reflective equilibrium, where you take your moral imperatives and heuristics and attempt to find the commonalities and consistencies they have with each other. It’s far from perfect, but I think that this method has produced useful results in the past.
Eliezer has proposed what is essentially a souped up version of reflective equilibrium called Coherent Extrapolated Volition. He has argued, however, that the primary use of CEV is in designing AIs that won’t want to kill us, and that attempting to extrapolate other people’s volition is open to corruption, as we could easily fall to the temptation to extrapolate it to something that personally benefits us.
Does there exist some objective method whereby you, I, a Pebblesorter, and a Paperclipper can all independently arrive at the same definition for what is moral (given that the Pebblesorter and Paperclipper will almost certainly promptly ignore the result)?
Again, we could probably get closer through reflective equilibrium, and by critiquing the methods and results of each other’s reflections. If you somehow managed to get a Pebblesorter or a Paperclipper to do it too, they might generate similar results, although since they don’t intrinsically care about the equation you would probably have to give them some basic instructions before they started working on the problem.
I had thought that you were proposing that we find that equation by summing across the moral values and imperatives of humanity as a whole—excluding the psychopaths.
If we assume that most humans care about acting morally, doing research about what people’s moral imperatives are might be somewhat helpful, since it would allow us to harvest the fruits of other people’s moral reflections and compare them with our own. We can exclude sociopaths because there is ample evidence that they care nothing for morality.
Although I suppose, that a super-genius sociopath who had the basic concept explained to them might be able to do some useful work in the same fashion that a Pebblesorter or Paperclipper might be able to. Of course, the genius sociopath wouldn’t care about the results, and probably would have to be paid a large sum to even agree to work on the problem.
I think that probably evolution metaphorically “wrote” a desire to care about the equation in our heads because if humans care about what is good and right it makes it easier for them to cooperate and trust each other, which has obvious fitness advantages.
Hmmm. That which evolution has “written” into the human psyche could, in theory, and given sufficient research, be read out again (and will almost certainly not be constant across most of humanity, but will rather exist with variations). But I doubt that morality is all in out genetic nature; I suspect that most of it is learned, from our parents, aunts, uncles, grandparents and other older relatives; I think, in short, that morality is memetic rather than genetic. Though evolution still happens in memetic systems just as well as in genetic systems.
So how do we learn more about this moral equation that we care about? One common form of attempting to get approximations of it in philosophy is called reflective equilibrium, where you take your moral imperatives and heuristics and attempt to find the commonalities and consistencies they have with each other. It’s far from perfect, but I think that this method has produced useful results in the past.
Hmmm. Looking at the wikipedia article, I can expect reflective equilibrium to produce a consistent moral framework. I also expect a correct moral framework to be consistent; but not all consistent moral frameworks are correct. (A paperclipper does not have what I’d consider a correct moral framework, but it does have a consistent one).
If you start out close to a correct moral framework, then reflective equilibrium can move you closer, but it doesnt necessarily do so.
Eliezer has proposed what is essentially a souped up version of reflective equilibrium called Coherent Extrapolated Volition. He has argued, however, that the primary use of CEV is in designing AIs that won’t want to kill us, and that attempting to extrapolate other people’s volition is open to corruption, as we could easily fall to the temptation to extrapolate it to something that personally benefits us.
Hmmm. The primary use of trying to find the True Morality Equation, to my mind, is to work it into a future AI. If we can find such an equation, prove it correct, and make an AI that maximises its output value, then that would be an optimally moral AI. This may or may not count as Friendly, but it’s certainly a potential contender for the title of Friendly.
Again, we could probably get closer through reflective equilibrium, and by critiquing the methods and results of each other’s reflections. If you somehow managed to get a Pebblesorter or a Paperclipper to do it too, they might generate similar results, although since they don’t intrinsically care about the equation you would probably have to give them some basic instructions before they started working on the problem.
Carrying through this method to completion could give us—or anyone else—an equation. But is there any way to be sure that it necessarily gives us the correct equation? (A pebblesorter may actually be a very good help in resolving this question; he does not care about morality, and therefore does not have any emotional investment in the research).
The first thought that comes to my mind, is to have a very large group of researchers, divide them into N groups, and have each of these groups attempt, independently, to find an equation; if all of the groups find the same equation, this would be evidence that the equation found is correct (with stronger evidence at larger values of N). However, I anticipate that the acquired results would be N subtly different, but similar, equations.
But I doubt that morality is all in out genetic nature; I suspect that most of it is learned, from our parents, aunts, uncles, grandparents and other older relatives; I think, in short, that morality is memetic rather than genetic.
That’s possible. But memetics can’t build morality out of nothing. At the very least, evolved genetics has to provide a “foundation,” a part of the brain that moral memes can latch onto. Sociopaths lack that foundation, although the research is inconclusive as to what extent this is caused by genetics, and what extent it is caused by later developmental factors (it appears to be a mix of some sort).
Hmmm. Looking at the wikipedia article, I can expect reflective equilibrium to produce a consistent moral framework. I also expect a correct moral framework to be consistent; but not all consistent moral frameworks are correct.
Yes, that’s why I consider reflective equilibrium to be far from perfect. Depending on how many errors you latch onto, it might worsen your moral state.
Carrying through this method to completion could give us—or anyone else—an equation. But is there any way to be sure that it necessarily gives us the correct equation?
Considering how morally messed up the world is now, even an imperfect equation would likely be better (closer to being correct) than our current slapdash moral heuristics. At this point we haven’t even achieved “good enough,” so I don’t think we should worry too much about being “perfect.”
However, I anticipate that the acquired results would be N subtly different, but similar, equations.
That’s not inconceivable. But I think that each of the subtly different equations would likely be morally better than pretty much every approximation we currently have.
But memetics can’t build morality out of nothing. At the very least, evolved genetics has to provide a “foundation,” a part of the brain that moral memes can latch onto. Sociopaths lack that foundation, although the research is inconclusive as to what extent this is caused by genetics, and what extent it is caused by later developmental factors
That sounds plausible, yes.
Considering how morally messed up the world is now, even an imperfect equation would likely be better (closer to being correct) than our current slapdash moral heuristics. At this point we haven’t even achieved “good enough,” so I don’t think we should worry too much about being “perfect.”
Hmmm. Finding an approximation to the equation will probably be easier than step two; encouraging people worldwide to accept the approximation. (Especially since many people who do accept it will then promptly begin looking for loopholes; either to use or to patch them).
However, if the correct equation cannot be found, then this means that the Morality Maximiser AI cannot be designed.
However, if the correct equation cannot be found, then this means that the Morality Maximiser AI cannot be designed.
That’s true, what I was trying to say is that a world ruled by a 99.99% Approximation of Morality Maximizer AI might well be far far better than our current one, even if it is imperfect.
Of course, it might be a problem if we put the 99.99% Approximation of Morality Maximizer AI in power, then find the correct equation, only to discover that the 99AMMAI is unwilling to step down in favor of the Morality Maximizer AI. On the other hand, putting the 99AMM AI in power might be the only way to ensure a Paperclipper doesn’t ascend to power before we find the correct equation and design the MMAI. I’m not sure whether we should risk it or not.
Hmmm. To avoid Omelas, equality would have to be fairly heavily weighted; any finite weighting given to equality, however, will simply mean that Omelas is only possible given a sufficiently large population (by balancing the cost of the inequality with the extra happiness of the extra inhabitants).
Well, if we’re really going to take Omelas seriously as our test case, then presumably we also have to look at how much that “extra happiness” (or whatever else we’re putting in the plus column) is reduced by those who walk away from it, and by those who are traumatized by it, and so forth. It might turn out that increasing the population doesn’t help.
But that’s just a quibble. I basically agree: once we swallow the assumption that for some reason we neither understand nor can ameliorate, the happiness of the many ineluctably depends on the misery of the few, then a total-utilitarian approach either says that equality is the most important factor in utility (which is a problem like you describe), or endorses the few being miserable.
That’s quite an assumption to swallow, though. I have no reason to believe it’s true of the world I live in.
A weaker version that might be true of the world I actually live in is that concentrating utility-generating resources in fewer hands results in higher total utility-from-all-sources-other-than-equality (Ua) but more total-disutility-from-inequality (Ub). But it’s not quite as clear that our (Ua, Ub) preferences are lexicographic.
Well, if we’re really going to take Omelas seriously as our test case, then presumably we also have to look at how much that “extra happiness” (or whatever else we’re putting in the plus column) is reduced by those who walk away from it, and by those who are traumatized by it, and so forth. It might turn out that increasing the population doesn’t help.
Doubling the population should double the happiness; double the trauma; double the people who walk away. The end result should be (assuming a high enough population that the Law of Large Numbers is a reasonable heuristic) about twice the utility.
A weaker version that might be true of the world I actually live in is that concentrating utility-generating resources in fewer hands results in higher total utility-from-all-sources-other-than-equality (Ua) but more total-disutility-from-inequality (Ub). But it’s not quite as clear that our (Ua, Ub) preferences are lexicographic.
Consider the case of farmland; larger farms produce more food-per-acre than smaller farms. (Why? Because larger farms attract commercial farmers with high-intensity farming techniques; and they can buy better farming equipment with their higher profits). Now, in the case of farmland, the optimal scenario is not equality; you don’t want everyone to have the same amount of farmland, you want those who are good at farming to have most of it. (For a rather dramatic example of this, see the Zimbabwe farm invasions).
On the other hand, consider the case of food itself. Here, equality is a lot more important; giving one man food for a hundred while ninety-nine men starve is clearly a failure case, as a lot of food ends up going rotten and ninety-nine people end up dead.
So the optimal (Ua, Ub) ordering depends on exactly what it is that is being ordered; there is no universally correct ordering.
You seem to be assuming a form of utility that is linear with happiness, with trauma, with food-per-acre, with starving people, etc. I agree with you that if we calculate utility this way, what you say follows. It’s not clear to me that we ought to calculate utility this way.
Hmmm. There are other ways to calculate utility, yes, and some of them are very likely better than linear. But all of them should at least be monotonically increasing with increased happiness, lower trauma, etc. There isn’t some point of global happiness where you can say that global happiness above this level is worse than global happiness at this level, if all else remains constant. The increase may be smaller for higher starting levels of happiness, but it should be an increase.
Such a system can either be bounded above by a maximum value, which it approaches asymptotically (such that no amount of global happiness alone can ever be worth, say, ten billion utilions, but it can approach arbitrarily close to that amount), or it can be unbounded (in which case enough global happiness can counter any finite amount of negative effects). A linear system would be unbounded, and my comments above can be trivially changed to fit with any unbounded system (but not necessarily with a bounded system).
It’s not clear to me whether it should be a bounded or an unbounded system..
all of them should at least be monotonically increasing with increased happiness, lower trauma, etc
OK, so we agree that doubling the population doesn’t provide twice the utility, but you’re now arguing that it at least increases the utility (at least, up to a possible upper bound which might or might not exist).
This depends on the assumption that the utility-increasing aspects of increased population increase with population faster than the utility-decreasing aspects of increased population do. Which they might not.
...you know, it’s only after I read this comment that I realised that you’re suggesting that the utility-decreasing aspects may not use the same function as the utility-decreasing aspects. That is, what I was doing was mathematically equivalent to first linearly combining the separate aspects, and only then feeding that single number to a monotonically increasing nonlinear function.
Now I feel somewhat silly.
But yes, now I see that you are right. There are possible ethical models (example: bounded asymptotic increase for positive utility, unbounded linear decrease for negative utility) wherein a larger Omelas could be worse than a smaller Omelas, above some critical maximum size. In fact, there are some functions wherein an Omelas of size X could have positive utility, while an Omelas of size Y (with Y>X) could have negative utility.
If you’re going to do that, why not just directly use happiness and fulfillment?
I cannot create an entire ethical framework, for everyone to follow, on any basis, and expect that it will be able to hold up for the next thousand years. If I try, I will fail, and this is why: because people cheat. Many intelligent agents will poke at the rules, seeking a possible exploit thereof that enhances their success at the possible expense of their neighbours’ success. Over the next thousand years, there will be thousands, probably millions, of such intelligent agents hunting for, and attempting to exploit, flaws in the system; people who stick by the letter of the rule, and avoid the spirit of the rule. I cannot create an entire ethical framework, because I cannot outwit thousands or millions of future peoples’ attempts to find and exploit gaps and loopholes in my framework.
Hence, the best that I can do is to find a system that has already endured a period of field testing and that hasn’t broken yet; and perhaps attempt a small, incremental improvement (no more) in order to test that improvement.
What does that have to do with the situation at hand? Morality is an abstract division of actions into right and wrong, not some set of laws laid down by philosophers on the rest of the population. If you’re trying to work out what you mean by “morality” and use some criteria (such as something including happiness and fulfillment of populations which adopt that definition) to choose from a bunch of alternatives, then probably those criteria themselves are the most accurate definition of “morality” you could hope to find. I might add, in [almost] exactly the same way that a program which writes and then executes a program to add two numbers is, in fact, itself a program that adds two numbers.
You can write out your final definition in legalese later, if the situation calls for it.
What does that have to do with the situation at hand? Morality is an abstract division of actions into right and wrong, not some set of laws laid down by philosophers on the rest of the population.
Morality comes with an implicit rule; when it says that “this action is the right action to take in this situation”, then the implicit rule is “if you find yourself in this situation, take this action”. There is usually no Morality Policeman ready to administer punishment if the rule is not followed, and the choice to follow the rule or not remains; but the rule is there.
f you’re trying to work out what you mean by “morality” and use some criteria (such as something including happiness and fulfillment of populations which adopt that definition) to choose from a bunch of alternatives, then probably those criteria themselves are the most accurate definition of “morality” you could hope to find.
The difficulty is that I know that the algorithm that I am following is very likely not to fulfil the criteria in the very best possible way; merely in (more or less) the best possible way that they have been fulfilled in the past. If I simply list the criteria, then I falsely imply that the chosen system of morality is the best fit for those criteria; and I am trying to avoid that implication.
I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters’ ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?
I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters’ ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?
Preferring other people like Larry to be homosexual is adaptive for me. And it is the judgement by others (and the implicit avoidance of that through shame) that we are considering here. That said:
I think the metaphor misses something important here
Absolutely, and the entire line of reasoning relies on conveying the speaker’s own morality (“it is second-order ‘right’ to be homosexual”) on others without making it explicit.
Wait, does it matter that homosexuality is probably not adaptive?
That was the point of my comment. There is a large disanalogy between heterosexuality and 13-pebble heap preference (namely, the first highly adaptive, but the second has no apparent reason to be). Although, I’m not sure if that is enough to break the metaphor.
There are many properties homosexuality has but 11-pebble heap preference don’t, and vice versa. Why is evolutionary maladaptiveness worth pointing out, is my question.
There are many properties homosexuality has but 11-pebble heap preference don’t, and vice versa. Why is evolutionary maladaptiveness worth pointing out, is my question.
Well, if moral norms are the Nash equilibria that result from actual historical bargaining situations (that are determined largely by human nature and the ancestral environment), then it seems somewhat relevant. If moral norms are actually imperative sentences uttered by God, then it seems completely irrelevant. Etc...
I suppose whether or not the pebble-sorting metaphor is good depends on which meta-ethical theory is true. In other words, I’m agreeing with PhilGoetz; Example 2 and Example 3 are only in separate classes of meta-wants assuming a (far from universally shared) moral system.
Well, if moral norms are the Nash equilibria that result from actual historical bargaining situations
I would regard moral norms as useful heuristics for achieving morally good results, not as morality in and of itself.
I suppose whether or not the pebble-sorting metaphor is good depends on which meta-ethical theory is true.
I think that some sort of ethical naturalism (or “moral cognitivism” as Eliezer calls it) is correct, where “morally good” is somewhat synonymous with “helps people live lives full of positive values like love, joy, freedom, fairness, high challenge, etc.” There is still much I’m not sure of, but I think, that is probably pretty close to the meaning of right. In Larry’s case I would argue that homosexual relationships usually do help people live such lives.
Incidentally, it’s easier to sort pebbles into heaps of 11. The original pebblesorters valued larger heaps, but had a harder time determining their correctness.
That’s why I was careful to refer to them as 11-Pebble and 13-Pebble Favorers. They do value other sizes of pebble heaps, 11 and 13 are just the numbers they do most frequently. Or perhaps 11 and 13 are the heaps they like making in their personal time, but they like larger prime numbers for social pebble-sorting endeavors. The point is, I said they “favored” that size because I wanted to make sure that the ease of sorting the piles didn’t seem too relevant, since that would distract away from the central metaphor.
I suspect we’re doing some extrapolation here in order to distinguish these cases. I expect that if Mimi knew more about herself and the world, and thought more clearly, she would still want to not want heroin; while I expect that if Larry knew more about himself and the world, and thought more clearly, he would be likely to reject the system of belief that causes him to think homosexuality immoral.
Alternatively, after sobering up, Mimi might decide that experiencing heroine high makes her life so much more fulfilling, that the much shortened life expectancy of a heroine addict doesn’t seems to be a fair price to pay for it.
As usual it is all up to personal definition of utility.
Contra Cyan & Alicorn, I am inclined to go with PhilGoetz and “punt it off to your moral system, or your expected-value computations.”
Trying to change your homosexual desires will probably fail and create a lot of collatoral damage. I would guess that trying to change your desire for heroin is somewhat more likely to succeed, though I’m willing to consider the argument that heroin addicts should accept their addiction but attempt to minimize its harmful side effects.
Mimi grew up in a society that taught her that heroin is bad. Larry grew up in a society that taught him that homosexuality is bad. How do you tell the difference from their point of view?
I think it probably has something to do with the fact that Mimi (probably) wasn’t born addicted to heroin (and even if she was, we can point to the behavior that caused it), whereas the consensus seems to be that homosexuality is innate.
I’m Paul Erdos. I’ve been taking amphetamine and ritalin for 20-odd years to enhance my cognitive performance. In general I want to want these drugs, because they help me do good, important and enjoyable work, which is impossible for me without them.
I can stop wanting these drugs when I want to, like when my friend bet me $500 that I couldn’t. I wanted to win that bet, so I wanted not to want the drugs, so I stopped wanting them. Was that my only motivation?
Also, I don’t want others to want to want amphetamines just because I want to want amphetamines.
A while ago I took Euler’s place as the most prolific mathematician of all time.
It’s not about whether it’s okay, it’s about whether it’s “part of who he is” or an alien intrusion.
That doesn’t solve PhilGoetz’s example though. And in the original version of Larry, his parents might very well say that his revulsion at homosexual acts is “who he is” and his sexual feelings the “alien intrusion”. Are these concepts anything but a way of making disguised moral judgements? Is “who someone really is” just “who I would prefer them to be”?
Then again, another attitude to Larry is that his sexual feelings are who he really is, but that resisting them is a cross he has to bear. (I believe this is the Roman Catholic view.) So I don’t think the concept of authenticity solves these problems.
I’m really bothered by my inability to see how to distinguish between these two classes of meta-wants. I suppose you just punt it off to your moral system, or your expected-value computations.
Looking at it, I think that the difference is that Larry the Closet Homosexual probably doesn’t really have a second order desire to not be gay. What he has is a second order desire to Do the Right Thing, and mistakenly believes that homosexuality isn’t the Right Thing. So we naturally empathize with Larry, because his conflict between his first and second order desires is unnecessary. If he knew that homosexuality wasn’t wrong the conflict would disappear, not because his desires had changed, but because he had better knowledge about how to achieve them.
Mimi the Heroin Addict, by contrast, probably doesn’t want to want heroin because it obstructs her from obtaining other important life goals that she genuinely wants and approves of. If we were too invent some sort of Heroin 2.0 that lacked most of heroin’s negative properties (i.e. removing motivation to achieve your life goals, health problems) Mimi would probably be much less upset about wanting it.
What reasoning process did you use to determine his belief was mistaken? When and where does Larry live? What are his other terminal goals?
In the interests of avoiding introducing complications into the thought experiment, I assumed that Larry was, aside from his sexual orientation, a fairly psychologically normal human who had normal human terminal goals, like an interest in sex and romantic love. I also assumed, again to avoid complications (and from clues in the story) that he probably lived, like most Less Wrong readers and writers, in a First World liberal democracy in the early 21st century.
The reasoning process I used to determine his belief was mistaken was a consequentialist meta-ethic that produces the results “Consensual sex and romance are Good Things unless they seriously interfere with some other really important goal.” I assumed that Larry, being a psychologically normal human in a tolerant country, did not have any other important goals they interfered with. He probably either mistakenly believed that a supernatural creature of immense power existed and would be offended by his homosexuality, or mistakenly believed in some logically incoherent deontological set of rules that held that desires for consensual sex and romance somehow stop being Good Things if the object of those desires is of the same sex as the desirer.
Obviously if Larry lived in some intolerant hellhole of a country or time period it might be well to change his orientation to be bisexual or heterosexual so that he could satisfy his terminal goals of Sex and Romance without jeopardizing his terminal goals of Not Being Tortured and Killed. But that would be a second best the solution, the ideal solution would be to convince his fellows that their intolerance was unethical.
PhilGoetz wrote:
I am having trouble seeing a significant difference between that and what you’ve described. Mimi’s enabler could argue “human happiness is a Good Thing unless it seriously interferes with some other really important goal,” and then one would have to make the engineering judgment of whether heroin addiction and homosexuality fall on opposite sides of the “serious interference” line. Similarly, the illegality of heroin and the illegality of homosexuality seem similarly comparable; perhaps Mimi should convince her fellows that their intolerance of her behavior is unethical.
Let me try using an extended metaphor to explain my point: Remember Eliezer’s essay on the Pebblesorters, the aliens obsessed with sorting pebbles into prime-numbered heaps?
Let’s imagine a race of Pebblesorters that’s p-morality consists of sorting pebbles into prime-numbered heaps. All Pebblesorters have a second-order desire to sort pebbles into prime-numbered heaps, and ensure that others do so as well. In addition to this, individual Pebblesorters have first order desires that make them favor certain prime numbers more than others when they are sorting.
Now let’s suppose there is a population of Pebblesorters who usually favor pebble heaps consisting of 13 pebbles but occasionally a mutant is born that likes to make 11-pebble heaps best of all. However, some of the Pebblesorters who prefer 13-pebble heaps have somehow come to the erroneous conclusion that 11 isn’t a prime number. Something, perhaps some weird Pebblesorter versions of pride and self-deception, makes them refuse to admit their error.
The 13-Pebble Favorers become obsessed with making sure no Pebblesorters make heaps of 11 pebbles, since 11 obviously isn’t a prime number. They begin to persecute 11-Pebble Favorers and imprison or kill them. They declare that Sortulon Prime, the mighty Pebblesorter God that sorts stars into gigantic prime-numbered constellations in the sky, is horribly offended that some Pebblesorters favor 11 pebble piles and will banish any 11-Pebble Favorers to P-Hell, where they will be forced to sort pebbles into heaps of 8 and 9 for all eternity.
Now let’s take a look at an individual Pebblesorter named Larry the Closet 11-Pebble Favorer. He was raised by devout 13-Pebble Favorer parents and brought up to believe that 11 isn’t a prime number. He has a second order desire to sort pebbles into prime-numbered heaps, and a first order desire to favor 11-pebble heaps. Larry is stricken by guilt that he wants to make 11-pebble heaps. He knows that 11 isn’t a prime number, but still feels a strong first order desire to sort pebbles into heaps of 11. He wishes he didn’t have that first order desire, since it obviously conflicts with his second order desire to sort pebbles into prime numbered heaps.
Except, of course, Larry is wrong. 11 is a prime number. His first and second order desires are not in conflict. He just mistakenly thinks they are because his parents raised him to think 11 wasn’t a prime number.
Now let’s make the metaphor explicit. Sorting pebbles into prime-numbered heaps represents Doing the Right Thing. Favoring 13-pebble heaps represents heterosexuality, favoring 11-pebble heaps represents homosexuality. Heterosexual sex and love and homosexual sex and love are both examples of The Right Thing. The people who think homosexuality is immoral are objectively mistaken about what is and isn’t moral, in the same way the 13-Pebble Favorers are objectively mistaken about the primality of the number 11.
So the first and second order desires of Larry the Closet Homosexual and Larry the Closet 11-Pebble Favorer aren’t really in conflict. They just think they are because their parents convinced them to believe in falsehoods.
Again, I assumed that Mimi was a psychologically normal human who had normal human second order desires, like having friends and family, being healthy, doing something important with her life, challenging herself, and so on. I assumed she didn’t want to use heroin because doing so interfered with her achievement of these important second order desires.
I suppose Mimi could be a mindless hedonist whose second order desires are somehow mistaken about what she really wants, but those weren’t the inferences I drew.
Again, recall my mention of a hypothetical Heroin 2.0 in my earlier comment. It seems to me that if Heroin 2.0 was suddenly invented, and Mimi still didn’t want to use heroin, even though it no longer seriously interfered with her other important values, that she might be mistaken. Her second order desire might be a cached thought leftover from when she was addicted to Heroin 1.0 and she can safely reject it.
But I will maintain that if Larry and Mimi are fairly psychologically normal humans, that Mimi’s second order desire to stop using heroin is an authentic and proper desire, because heroin use seriously interferes with the achievement of important goals and desires that normal humans (like Mimi, presumably) have. Larry’s second order desire, by contrast, is mistaken, because it’s based on the false belief that homosexuality is immoral. Homosexual desires do not interfere with important goals humans have. Rather, they are an important goal that humans have (love, sex, and romance), it’s just that the objective of that goal is a bit unusual (same sex instead of opposite).
EDITED: To change some language that probably sounded too political and judgemental. The edits do not change the core thesis in any way.
We should point people to this whenever they’re like “What’s special about Less Wrong?” and we can be like “Okay, first, guess how Less Wrong would discuss a reluctant Christian homosexual. Made the prediction? Good, now click this link.”
I’m surprised you regarded it so highly. The flaws I noticed are located in a response to Ghatanathoah’s comment.
First, I would like to make one thing clear: I have absolutely nothing against homosexuals and in fact qualify as queer because my attractions transcend gender entirely. I call my orientation “sapiosexual” because it is minds that I am sexually attracted to, and good character, never mind the housing.
Stops at “pigheaded jerks”
downvotes
You know where this is going, oh yes, I am going right to fundamental attribution error and political mindkill.
The parents are deemed “pigheaded jerks”—a perception of their personality.
Larry the homosexual, convinced by the exact same reasoning, is given something subtly different—an attack on his behavior—“he gullibly believed them” and you continue with “They (the Larrys) just think they are because their parents fed them a load of crap.” attributing his belief to the situation that Larry is in.
Do you think Larry’s grandparents didn’t teach Larry’s parents the same thing? And that Larry’s great grandparents didn’t teach it to Larry’s grandparents?
This was a “good solid dig” at the other side.
I upvoted despite this. If you overlook that one problem, everything else is gold. That single flawed sentence does not effect the awesome of the other 14 paragraphs, as it does not contribute to the conclusion.
My experience of it was more like:
“Oh, this is nice and organized… Still orderly… Still orderly… OHMYSPAGHETTIMONSTER I DID NOT JUST READ THAT!”
To me, it was a disappointment. Like if I were eating ice cream and then it fell to the ground.
If Eliezer is going to praise it like it’s the epitome of what LessWrong should be, then it should be spotless. Do you agree?
You make an excellent point. I will edit my post to make it sound less political and judgemental.
I am charmed by your polite acknowledgement of the flaw and am happy to see that this has been updated. Thanks for letting me know that pointing it out was useful. :)
I think you’re looking at this discussion from the wrong angle. The question is, “how do we differentiate first-order wants that trump second-order wants from second-order wants that trump first-order wants?” Here, the order only refers to the psychological location of the desire: to use Freudian terms, the first order desires originate in the id and the second order desires originate in the superego.
In general, that is a complicated and difficult question, which needs to be answered by careful deliberation- the ego weighing the very different desires and deciding how to best satisfy their combination. (That is, I agree with PhilGoetz that there is no easy way to distinguish between them, but I think this is proper, not bothersome.)
Some cases are easier than others- in the case of Sally, who wants to commit suicide but wants to not want to commit suicide, I would generally recommend methods of effective treatment for suicidal tendencies, not the alternative. But you should be able to recognize that the decision could be difficult, at least for some alteration of the parameters, and is the alteration is significant enough it could swing the other way.
There is also another factor which clouds the analysis, which is that the ego has to weigh the costs of altering, suppressing, or foregoing one of the desires. It could be that Larry has a twin brother, Harry, who is not homosexual, and that Harry is genuinely happier that Larry is, and that Larry would genuinely prefer being Harry to being himself; he’s not mistaken about his second-order want.
However, the plan to be (or pretend to be) straight is much more costly and less likely to succeed than the plan to stop wanting to be straight, and that difference in costs might be high enough to determine the ego’s decision. Again, it should be possible to imagine realistic cases in which the decision would swing the other way. (Related.)
It’s also worth considering how much one wants to engage in sour grapes thinking- much of modern moral intuitions about homosexuality seem rooted in the difficulty of changing it. (Note Alicorn’s response. Given that homosexuality is immutable, then plans to change homosexuals are unlikely to succeed, and they might as well make the best of their situation. But I hope it’s clear that, at its root, this is a statement about engineering reality, not moral principles- if there were a pill that converted homosexuals to heterosexuals, then the question of how society treats homosexuals would actually be different, and if Larry asked you to help him make the decision of whether or not to take the pill, I’m sure you could think of some things to write in the “pro” column for “take the pill” and in the “con” column for “don’t take the pill.”
Why I said this is worth considering is that, as should be unsurprising, two wants conflict. Often, we don’t expect the engineering reality to change. Male homosexuality is likely to be immutable for the lifetimes of the ones that are currently alive, and it’s more emotionally satisfying to declare that homosexual desires don’t conflict with important goals than reflect on the tradeoffs that homosexuals face that heterosexuals don’t. Doing so, however, requires a sort of willful blindness, which may or may not be worth the reward gained by engaging in it.
I don’t deny that there may be some good reasons to prefer to be heterosexual. For instance, imagine Larry lives in an area populated by very few homosexual and bisexual men, and moving somewhere else is prohibitively costly for some reason. If this is the case, then Larry may have a rational second-order desire to become bisexual or heterosexual, simply because doing so would make it much easier to find romantic partners.
However, I would maintain that the specific reason given in Alicorn’s orignal post for why Larry desires to not be homosexual is that he is confused about the morality of homosexuality and is afraid he is behaving immorally, not because he has two genuine desires that conflict.
I find it illuminating to compare intuitions about homosexuality to intuitions about bisexuality. If homosexual relationships were really inferior to heterosexual ones in some important way then it would make sense to encourage bisexual people to avoid homosexual relationships and focus on heterosexual ones. This seems wrong to me however, if I was giving a bisexual person relationship advice I think the good thing to do would be advise them to focus on whoever is most compatible with them regardless of sex.
I think you are probably right, this is proper. I think I may feel biased in favor of second order desires because right now it seems like in my current life I have difficulty preventing my first order desires from overriding them. But if I think about it, it seems like I have many first order desires I cherish and would really prefer to avoid changing.
While the Freudian description is accurate relative to sources, I struggle to order them. I believe it is an accumulated weighting that makes one thought dominate another. We are indeed born with a great deal of innate behavioral weighting. As we learn, we strengthen some paths and create new paths for new concepts. The original behaviors (fight or flight, etc.) remain.
Based on this known process, I conjecture that experiences have an effect on the weighting of concepts. This weighting sub-utility is a determining factor in how much impact a concept has on our actions. When we discover fire burns our skin, we don’t need to repeat the experience very often to weigh fire heavily as something we don’t want touching our skin.
If we constantly hear, “blonde people are dumb,” each repetition increases the weight of this concept. Upon encountering an intelligent blond named Sandy, the weighting of the concept is decreased and we create a new pattern for “Sandy is intelligent” that attaches to “Sandy is a person” and “Sandy is blonde.” If we encounter Sandy frequently, or observe many intelligent blonde people, the weighting of the “blonde people are dumb” concept is continually reduced.
Coincidentally, I believe this is the motivation behind why religious leaders urge their followers to attend services regularly, even if subconsciously. The service maintains or increases weighting on the set of religious concepts, as well as related concepts such as peer pressure, offsetting any weighting loss between services. The depth of conviction to a religion can potentially be correlated with frequency of religious events. But I digress.
Eventually, the impact of the concept “blonde people are dumb” on decisions becomes insignificant. During this time, each encounter strengthens the Sandy pattern or creates new patterns for blondes. At some level of weighting for the “intelligent” and “blonde” concepts associated to people, our brain economizes by creating a “blond people are intelligent” concept. Variations of this basic model is generally how beliefs are created and the weights of beliefs are adjusted.
As with fire, we are extremely averse to incongruity. We have a fundamental drive to integrate our experiences into a cohesive continuum. Something akin to adrenaline is released when we encounter incongruity, driving us to find a way to resolve the conflicting concepts. If we can’t find a factual explanation, we rationalize one in order to return to balanced thoughts.
When we make a choice of something over other things, we begin to consider the most heavily weighted concepts that are invoked based on the given situation. We work down the weighting until we reach a point where a single concept outweighs all other competing concepts by an acceptable amount.
In some situations, we don’t have to make many comparisons due to the invocation of very heavily weighted concepts, such as when a car is speeding towards us while we’re standing in the roadway. In other situations, we make numerous comparisons that yield no clear dominant concept and can only make a decision after expanding our choice of concepts.
This model is consistent with human behavior. It helps to explain why people do what they do. It is important to realize that this model applies no division of concepts into classes. It uses a fluid ordering system. It has transient terminal goals based on perceived situational considerations. Most importantly, it bounds the recursion requirements. As the situation changes, the set of applicable concepts to consider changes, resetting the core algorithm.
From what I’ve heard, the typical response to believing that blond people are dumb and observing that blond Sandy is intelligent is to believe that Sandy is an exception, but blond people are dumb.
Most people are very attached to their generalizations.
Quite right about attachment. It may take quite a few exceptions before it is no longer an exception. Particularly if the original concept is regularly reinforced by peers or other sources. I would expect exceptions to get a bit more weight because they are novel, but no so much as to offset higher levels of reinforcement.
Your example does an exemplery job of explaining your viewpoint on Larry’s situation. To explain the presumed viewpoint of Larry’s parents on his situation requires merely a very small change; replacing all occurrances of the number 11 with the number 9.
How do you define objective morality? I’ve heard of several possible definitions, most of which conflict with each other, so I’m a little curious as to which one you’ve selected.
I’m not sure I understand you, do you mean that a more precise description of Larry’s parent’s viewpoint is that the Pebblesorter versions of them think 11 and 9 are the same numbers? Or are you trying to explain how a religious fundamentalist would use the Pebblesorter metaphor if they were making the argument.
I define morality as being a catch-all term to describe what are commonly referred to as the “good things in life,” love, fairness, happiness, creativity, people achieving what they want in life, etc. So something is morally good if it tends to increase those things. In other words, “good” and “right” are synonyms. Morality is objective because we can objectively determine whether people are happy, being treated fairly, getting what they want out of life, etc. In Larry’s case having a relationship with Ted-the-Next-Door neighbor would be the morally right thing to do because it would increase the amount of love, happiness, people-getting what they want, etc. in the world.
I think the reason that people have such a problem with the idea of objective morality is that they subscribe, knowingly or not, to motivational internalism. That is, they believe that moral knowledge is intrinsically motivating, simply knowing something is right motivates someone to do it. They then conclude that since intrinsically motivating knowledge doesn’t seem to exist, that morality must be subjective.
I am a motivational externalist, so I do not buy this argument. I believe that that people are motivated to act morally by our conscience and moral emotions (i.e. compassion, sympathy). If someone has no motivation to act to increase the “good things in life” that doesn’t mean morality is subjective, that simply means that they are a bad person. People who lack moral emotions exist in real life, and they seem to lack any desire to act morally at all, unless you threaten to punish them if they don’t.
The idea of intrinsically motivating knowledge is pretty scary if you think about it. What if it motivated you to kill people? Or what if it made you worship Darkseid? The Anti-Life equation from Final Crisis works pretty much exactly the way motivational internalists think moral knowledge does, except that instead of motivating people to care about others and treat people well, it instead motivates them to serve evil pagan gods from outer space.
Yes, exactly. Larry’s parents’ do not believe that they are mistaken, and are not easily proved mistaken.
That’s a good definition, and it avoids most of the obvious traps. A bit vague, though. Unfortunately, there is a non-obvious trap; this definition leads to the city of Omelas, where everyone is happy, fulfilled, creative… except for one child, locked in the dark in a cellar, starved; one child on whose suffering the glory of Omelas rests. Saving the child decreases overall happiness, health, achievement of goals, etc., etc. Despite all this, I’d still think that leaving the child locked away in the dark is a wrong thing. (This can also lead to Pascal’s Mugging, as an edge case)
In my case, it’s because every attempt I’ve seen at defining an objective morality has potential problems. Given to you by an external source? But that presumes that the external source is not Darkseid. Written in the human psyche? There are some bad things in the dark corners of the human psyche. Take whatever action is most likely to transform the world into a paradise? Doesn’t usually work, because we don’t know enough to always select the correct actions. Do unto others as you would have them do unto you? That’s a very nice one—but not if Bob the Masochist tries to apply it.
Of course, subjective morality is no better—and is often worse (mainly because a society in general can reap certain benefits from a shared idea of morality).
What does seem to work is to pick a society whose inhabitants seem happy and fulfilled, and trying to use whatever rules they use. The trouble with that is that it’s kludgy, uncertain, and could often do with improvement (though it’s been improved often enough in human history that many—not all, but many—obvious ‘improvements’ aren’t in practice).
Aside from its obvious artificiality, and despite the fact that all our instincts cry out against it, it’s not at all clear to me that there are any really good reasons to reject the Omelasian solution. This is of course a fantastically controversial position (just look at the response to Torture vs. Dust Specks, which might be viewed as an updated and reframed version of the central notion of The Ones Who Walk Away From Omelas), but it nonetheless seems to be a more or less straightforward consequence of most versions of consequential ethics.
As a matter of fact, I’m inclined to view Omelas as something between an intuition pump and a full-blown cognitive exploit: a scenario designed to leverage our ethical heuristics (which are well-adapted to small-scale social groups, but rather less well adapted to exotic large-scale social engineering) in order to discredit a viewpoint which should rightfully stand or fall on pragmatic grounds. A tortured child is something that hardly anyone can be expected to think straight through, and trotting one out in full knowledge of this fact in order to make a point upsets me.
There’s a real world analogue to Omelas. The UK (like other countries) has a child protection system, intended to minimize abuse & neglect of children. The state workers (health visitors, social workers, police officers, hospital staff, etc.) who play roles in the system can try to intervene when the apparent risk of harm to a child reaches some threshold.
If the threshold is too low, the system gratuitously interferes with families’ autonomy, preventing them from living their lives in peace. If the threshold is too high, the system fails to do its job of preventing guardians from tormenting or killing their children. Realistically, a trade-off is inevitable, and under any politically feasible threshold “some children will die to preserve the freedom of others”, as the sociologist Robert Dingwall put it. So the UK’s child protection system takes the Omelasian route.
The real life situation is less black & white than Omleas, but it looks like the same basic trade-off in a non-artificial setting. I wonder whether people’s intuitions about Omelas align with their intuitions about the real life child protection trade-off (and indeed whether both align with society’s revealed collective preference).
I’d agree that these are consequentially similar, but I don’t think they’re psychologically similar at all. There’s an element of exploitation in Omelas that isn’t present in social services: state workers are positioned as protecting children from an evil unrelated to the state, while Omelas is cast as willfully perpetrating an evil in order to ensure its own prosperity. People tend to think of moral culpability in terms of blame, and although some blame might attach itself to social workers for failing to stop abuses that they might prevent with more intrusive intervention thresholds, it’s much diluted by the vastly more viscerally appalling culpability carried by actual abusers. Omelas offers no subjects for condemnation other than the state apparatus and the citizens supporting it.
On top of that, intrusive child protection services have very salient failings: most parents (and most children) would find government intrusion into their family lives extremely unpleasant, unpleasant enough to fear and take political action to avoid. Meanwhile, the consequences of no longer torturing Omelas’ sacrificial lamb are unspecified and thus about as far from salient (re-entrant?) as it’s possible to get. Even in a hypothetical fully specified Omelas where we could point to a chain of effects, I’d expect that chain to be a lot longer and harder to follow, and its endpoints hence less emotionally weighty.
Link to me mentioning both Omelas and another “eternally tortured child” short story, SCP-231 (potentially highly distressing so I’m not hotlinking it), as an intuition pump against Mencius Moldbug’s “Patchwork” proposal (the “strong”/”total” vision of Patchwork, with absolute security of sovereigns) along very similar lines of analogy, over in the Unqualified Reservations comments.
Disagree SCP-231 is bad source of intuition because it is crafted to be torture & horror porn.
There are many, many things in the similarly exploitative franchise known as “Real Life” that also appear to be crafted as “torture & horror porn”. So I don’t see the problem with linking to a fictionalized version.
I dare say that any story without elements that induce horror and revulsion in a reader would be an inadequate source of intuition for considering the most shocking aspects of our own world… or the ethics of knowingly creating a system which offers absolute security indiscriminately to those who would create such nightmares and those who’d seek to prevent them.
Example of a victim testimony 1: - trigger warnings for extreme child abuse, rape, pedophilia and psychological damage.
Example of a victim testimony 2: - all of the above, except even more outspoken descriptions of the author’s mental anguish. (NSFanywhere. The main blog has… images… that are more gore than extreme porn; don’t look unless you’re massively desensitized.)
Want some amnesiacs yet? You might be able to forget those stories faster if you don’t think about the fact that something similar must be happening somewhere in your country, probably in your city, at this very moment. Oops, too late!
(Again, sorry for the confrontational tone and such—I wanted to hammer home the point that sometimes it’s the violently emotional reaction to an objectively terrible problem that would be true to your desires, and trying to stay “detached” and “reasonable” would be self-deception. See: deathism.)
Indeed. But in the context of the discussion the story primes you to live one kind of horror and not another when making trade offs between the two. This is why I objected to it.
I struggle with conceiving wanting to want, or decision making in general, as a tiered model. There are a great many factors that modify the ordering and intensity of utility functions. When human neurons fire they trigger multiple concurrent paths leading to a set of utility functions. Not all of the utilities are logic-related.
I posit that our ability to process and reason is due to this pattern ability and any model that will approximate human intelligence will need to be more complex than a simple linear layer model. The balance of numerous interactive utilities combine to inform decision making. A multiobjective optimization model, such as PIBEA, is required.
I’m new to LW, so I can’t open threads just yet. I’m hoping to find some discussions around evolutionary models and solution sets relative to rational decision processing.
I sort of agree with your comment, but you should’ve probably posted it as a reply to the main post, not to this specific comment of mine, as the topics seem to be somewhat apart.
(BTW you can open threads in Discussion just fine seeing as you’re at 10 karma—go ahead!)
Disagree. Do look, even if it hurts. Especially if it hurts a lot (if it doesn’t hurt at all, there’s a lot more wrong to fix first). Update your model of reality, avoid availability and (dis)confirmation bias. Face reality head-on. That kind of thing.
All due respect, but I don’t think there is any imperative to view material that will cause serious psychological harm. If the price of knowledge is your right hand you are not required to pay it. The most relevant example would be showing hardcore pornography to small children—it may be knowledge of a kind, but it does more harm than good.
EDIT: Of course, I read them anyway. Very, ah … deep.
I only said that in regards to the many photos of staged degradation, multilation, self-harm, etc on the second blog—they’re cringe-inducing yet presumably taken in a consensual way (for a loose definition of consent), and thus have a worse “unpleasantness-to-facing-reality” ratio than the textual descriptions of real abuse & its consequences. I quite agree that reading those shocking descriptions is epistemologically and morally imperative.
Good point. If there are written descriptions available, they’re a better weighted alternative.
It would surprise me if they did, given that Omelas was constructed as an intuition pump.
Omelas is a cognitive exploit, yes. That’s really the point—it forces people to consider how appropriate their heuristics really are. Some people would make Omelas if they could; some wouldn’t, for the sake of the one child. A firm preference for either possibility can be controversial, partially because there are good reasons for both states and partially because different ethical heuristics get levered in different directions. (A heuristic that compares the number of people helped vs. the number hurt will pull one way; a heuristic that says “no torture” will pull the other way).
Personally, on reading the story, I decided immediately that not only would I not walk away from Omelas (which solves nothing anyway,) I was fully in favor of the building of Omelases, provided that even more efficient methods of producing prosperity were not forthcoming.
The prevention of dust specks may vanish into nothing in my intuitive utility calculations, but it immediately hits me that a single tortured child is peanuts besides the cumulative mass of suffering that goes on in our world all the time. With a hundred thousand dollars or so to the right charity, you could probably prevent a lot more disutility than that of the tortured child. If for that money I could either save one child from that fate, or create a city like Omelas minus the sacrifice, then it seems obvious to me that creating the city is a better bargain.
That’s true, but I think that human values are so complex that any attempt to compress morality into one sentence is pretty much obligated to be vague.
One rather obvious rejoinder is that there are currently hundreds, if not thousands of children who are in the same state as the unfortunate Omelasian right now in real life, so reducing the number to just one child would be a huge improvement. But you are right that even one seems too many.
A more robust possibility might be to add “equality” to the list of the “good things in life.” If you do that then Omelas might be morally suboptimal because the vast inequality between the child and the rest of the inhabitants might overwhelm the achievement of the other positive values. Now, valuing equality for its own sake might add other problems, but these could probably be avoided if you were sufficiently precise and rigorous in defining equality.
I think the best explanation I’ve seem is something like the metaethics Eliezer espouses, which is (if I understand them correctly), that morality is a series of internally consistent concepts related to achieving what I called “the goods things in life,” and that human beings (those who are not sociopaths anyway) care a lot about these concepts of wellbeing and want to follow and fulfill them.
In other words, morality is like mathematics in some ways, it generates consistent answers(on the topic of people’s wellbeing) that are objectively correct. But it is not like the Anti-Life Equation because it is not intrinsically motivating. Humans care about morality because of our consciences and our positive emotions, not because it is universally compelling.
To put it another way, I think that if you were to give a superintelligent paperclipper a detailed description of human moral concepts and offered to help it make some more paperclips if it elucidated these concepts for you, that it would probably generate a lot of morally correct answers. It would feel no motivation to obey these answers of course, since it doesn’t care about morality, it cares about making paperclips.
This is a little like morality being “embedded in the human psyche” in the sense that the desire to care about morality is certainly embedded in their somewhere (probably in the part we label “conscience”). But it is also objective in the sense that moral concepts are internally consistent independent of the desires of the mind. To use the Pebblesorter metaphor again, caring about sorting pebbles into prime numbered heaps is “embedded in the Pebblesorter psyche,” but which numbers are prime is objective.
That’s certainly true, but that simply means that humans are capably of caring about other things besides morality, and these other things that people sometimes care about can be pretty bad. This obviously makes moral reasoning a lot harder, since it’s possible that one of your darker urges might be masquerading as a moral judgement. But that just means that moral reasoning is really hard to do, it doesn’t mean that it’s wrong in principle.
Vague or flawed. Given those options, I think I’d prefer vague.
I agree completely. If I had any idea how Omelas worked, I might be tempted to try seeing if any of those ideas could be used to improve current societies.
Hmmm. To avoid Omelas, equality would have to be fairly heavily weighted; any finite weighting given to equality, however, will simply mean that Omelas is only possible given a sufficiently large population (by balancing the cost of the inequality with the extra happiness of the extra inhabitants).
Personally, I think that valuing equality itself is a good idea, if mixed in with a suitable set of other values; one possible failure mode for overvaluing equality is an equality of wretchedness, a state of “we’re all equal because we all have nothing and no hope” (this is counteracted by providing suitable weights for the other good things, like happiness and the freedom to try to achieve goals (but what about the goal of vengeance? For a real slight? An imagined slight?)).
An example of a society failing due to an over-reliance on equality, is France shortly after the French Revolution.
I think that I should read through that entire sequence in the near future. I’d like to see his take on metaethics.
Huh. I think we’re defining ‘morality’ slightly differently here.
My definition of ‘morality’ would be ‘a set of rules, decided by some system, such that one can feed in a given action and (usually) get out whether that action was a good or a bad action’. Implicit in that definition is the idea that two people may disagree on what those rules actually are—that there might be better or worse moralities, and that therefore the answers given by a randomly chosen morality need not be objectively correct.
To take an example; certain ancient cultures may have had the belief that human sacrifice was necessary, on Midwinter’s Day, to persuade summer to come back and let the crops grow. In such a culture, strapping someone down and killing them in a particularly painful way may have been considered the right thing to do; and a member of that society would argue for it on the basis that the tribe needs the crops to grow next year (and, if selected, might even walk up voluntarily to be killed). In his morality, these annual deaths are a good thing, because they make the crops grow; in my morality, these annual deaths are a bad thing, and moreover, they don’t make the crops grow.
For what it’s worth, I do agree with you that getting the result out of a moral system does not, in and of itself, force an intrinsic motivation to follow that course of action; people can be trained from a young age to feel that motivation, and many people are, but there’s really no reason to assume that it is always there.
If there is an objectively correct morality, that can apply to all situations, then I don’t know what it is—my current system of morality (based heavily on the biblical principal of ‘Love thy neighbour’) covers many situations, but is not good at the average villain’s sadistic choice (where I can save the lives of group A or group B but not both)
Hmmm. A lot of the darkness in the human psyche can be explained in this manner; but I’d think that there are other parts which cannot be explained in this way (when a person goes out of his way to hurt someone that he’ll likely never see again, for example). A lot of these, I’d think, are attributable to a lack of empathy; a person who sees other people as non-people (or as Not True People, for some self-including definition of True People).
I think a possible solution would be to have equality and the other values have diminishing returns relative to each other. So in a society with a lot of other good things there is a great obligation to increase equality, whereas in a society with lots of suffering people it’s more important to do whatever it takes raise the general level of happiness and not worry as much about equality. So a place as wondrous as Omelas would have a great obligation to help the child.
I think one possible way to frame equality to avoid this is to imagine, metaphorically, that positive things give a society “morality points” and negative things give it “negative morality points.” Then have it so that a positive deed that also decreases equality gets “extra points,” while a negative deed that also exasperates inequality gets “extra negative points.” So in other words, helping the rich isn’t bad, it’s just much less good than helping the poor.
This also avoids another failure mode: Imagine an action that hurts every single person in the world, and hurts the rich 10 times as much as it hurts the poor. Such an action would increase equality, but praising it seems insane. Under the system I proposed such an action would still count as “bad,” though it would be a bit less bad than a bad action that also increased inequality.
I don’t think that’s that different from what I’m saying, I may be explaining it poorly. I do think that morality is essentially like a set of rules or an equation that one uses to evaluate actions. And I consider it objective in that the same equation should produce the same result each time an identical action is fed into it, regardless of what entity is doing the feeding. Then it is up to our moral emotions to motivate us to take actions the equation would label as “good.”
Describing it like that sounds a bit clinical though, so I’d like to emphasize that moral rules and equations, are ultimately about people’s wellbeing and increasing the good things in life. If you feed an action that improves these values into a rule-set and it comes out labelled “bad” then those rules probably don’t even deserve to be called morality, they are some other completely different concept.
This relates to Eliezer’s metaethics again, he basically describes morality as an equation or “fixed computation” related to wellbeing that is so complex that it’s impossible to wrap your mind about it, so you have to work in approximations. So what you would label a “better” morality is one that more closely resembles the “ideal equation.”
It seems to me that this is more a disagreement about certain facts of nature than about morality per se. It seems to me that if there really was some sort of malevolent supernatural entity that wouldn’t let summer come unless you made sacrifices to it, and it was impossible to stop such an entity, that sacrificing to it might be the only option left. If the choice is “everyone dies of starvation” vs. “one person dies from being sacrificed, everyone else lives” it seems like any worthwhile set of moral rules would label the second option as the better one (though it would not be nearly as good as somehow stopping the entity). The reason that sacrificing people is bad is because such entities do not exist, so such a sacrifice tortures someone, but doesn’t save anyone elses’ life.
I think the problem is that an objectively correct set of moral rules that could perfectly evaluate any situation would be so complicated no one would be able to use it effectively. Even if we obtained such a system we would have to use crude approximations until we managed to get a supercomputer big enough to do the calculations in a timely manner.
I count empathy as one of the “moral emotions” that motivates people to act morally. So a lack of empathy would be a type of lack of motivation towards moral behavior.
That seems to work very well. So the ethical weight of a factor can be proportional to the reciprocal thereof (perhaps with a sign change). Then, for any number of people, there is a maximum happiness-factor that the equation can produce.
So. This can be used to make an equation that makes Omelas bad for any sized population. But not everyone agrees that Omelas is bad in the first place; so is that necessarily an improvement to your ethical equation?
That failure mode can also be dealt with by combining equality with other factors, such as not being hurt. (The relative weightings assigned to these factors would be important, of course).
That seems like a reasonable definition; my point is that not everyone uses the same equation.
Hmmm. You’re right—that was a bad example. (I don’t know if you’re familiar with the Chanur series, by C. J. Cherryh? I ask because my first thought for a better example came straight out of there—she does a god job of presenting alien moralities)
Let me provide a better one. Consider Marvin, and Fred. Marvin’s moral system considers the total benefit to the world of every action; but he tends to weight actions in favour of himself, because he knows that in the future, he will always choose to do the right thing (by his morality) and thus deserves ties broken in his favour.
Fred’s moral system entirely discounts any benefits to himself. He knows that most people are biased to themselves, and does this in an attempt to reduce the bias (he goes so far as to be biased in the opposite direction).
Both of them get into a war. Both end up in the following situation:
Trapped in a bunker, together with one allied soldier (a stranger, but on the same side). An enemy manages to throw a grenade in. The grenade will kill both of them, unless someone leaps on top of it, in which case it will only kill that one.
Fred leaps on top of the grenade. His morality values the life of the stranger over his own, and he thus acts to save the stranger first.
Marvin throws the stranger onto the grenade. His morality values his own life over a stranger who might, with non-trivial probability, be a truly villainous person.
Here we have two different moralities, leading to two different results, in the same situation.
That is worth keeping in mind. Of course, if such a system is found, we could feed in dozens of general situations in advance—and if in a tough situation, then after resolving it one way or another, we could feed that situation into the computer and find out for future reference which course of action was correct (that eliminates a lot of the time constraint).
That’s true, the question is, how often is this because people have totally different values, and how often is it that they have extremely similar “ideal equations,” but different “approximations” of what they think that equation is. I think for sociopaths, and other people with harmful ego-syntonic mental disorders it’s probably the former, but its more often the later for normal people.
Eliezer has argued that it is confusing and misleading to use the word “morality” to refer to codes of behavior entities possess that have nothing to do with improving people’s wellbeing, making the world a happier, fairer, freer place, and similar concepts. He argues that creatures like the Pebblesorters do not care about morality at all, they care about sorting pebbles and calling sorting pebbles a type of “morality” confuses two separate concepts.
It sounds to me like Fred and Marvin both care about achieving similar moral objectives, but have different ideas about how to go about it. I’d say that again, which moral code is better can only be determined by trying to figure out which one actually does a better job of achieving moral goals. “Moral progress” can be regarded as finding better and better heuristics to achieve those moral goals, and finding a closer representation of the ideal equation.
Again, I think I agree with Eliezer that a truly alien code of behavior, like that exhibited by sociopaths, and really inhuman aliens like the Pebblesorters or paperclippers, should maybe be referred to by some word other than morality. This is because since the word “morality” usually refers to doing things like making the world a happier place and increasing the positive things in life. So if we refer to the behavior code of a creature that cares nothing for doing those things as “morality,” we will give the subconscious impression that that creature really does care about doing good and simply disagrees about how to go about it. This isn’t correct, a sociopaths and paperclippers doesn’t care about other people at all, so we shouldn’t give the impression they do.
I am less sure about whether the term “morality” should be used to refer to the behavior codes of aliens that care about some of the same positive things that normal humans do, but also differ in important ways, like the Babyeaters and Super-Happy-People. Maybe we could call it “semi-morality?”
Sorry, the only Cherryh I’ve read is “The Scapegoat.” I thought it gave a good impression of how alien values would look to humans, but wish it had given some more ideas about what it was that made elves think so differently.
I’d say sometimes A, and sometimes B. But I think that’s true even in the absence of mental disorders; I don’t think that the “ideal equation” necessarily sits somewhere hidden in the human psyche.
That is valid, as long as both systems have the same goals. Marvin’s system includes the explicit goal “stay alive”, more heavily weighted then the goal “keep a stranger alive”; Fred’s system explicitly entirely excludes the goal “stay alive”.
If two moral systems agree both on the goals to be achieved, and the weightings to give those goals, then they will be the same moral system, yes. But two people’s moral systems need not agree on the underlying goals.
Well, to be fair, in a Paperclipper’s mind, paperclips are the positive things in life, and they certainly make the paperclipper happier. I realise that’s probably not what you intended, but the phrasing may need work.
Which really feeds into the question of what goals a moral system should have. To the Babyeaters, a moral system should have the goal of eating babies, and they can provide a lot of argument to support that point—in terms of improved evolutionary fitness, for example.
I think that we can agree that a moral system’s goals should be the good things in life. I’m less certain that we can agree on what those good things necessarily are, or on how they should be combined relative to each other. (I expect that if we really go to the point of thoroughly dissecting what we consider to be the good things in life, then we’ll agree more than we disagree; I expect we’ll be over 95% in agreement, but not quite 100%. This is what I generally expect for any stranger).
For example, we might disagree on whether it is more important to be independant in our actions, or to follow the legitimate instructions of a suitably legitimate authority.
Hmmm. I haven’t read that one.
It’s not that I think there’s literally a math equation locked in the human psyche that encodes morality. It’s more like there are multiple (sometimes conflicting) moral values and methods for resolving conflicts between them and that the sum of these can be modeled as a large and complicated equation.
You gave me the impression that Marvin valued “staying alive” less as an end in itself, and more as a means to achieve the end of improving the world. in particular when you said this:
This is actually something that bothers me in fiction when a character who is superhumanly good and power (i.e. Superman, the Doctor) risks their lives to save a relatively small amount of people. It seems short-sighted of them to do that since they regularly save much larger groups of people and anticipate continuing to do so in the future, so it seems like they should preserve their lives for those people’s sakes.
If you define “the good things in life” as “whatever an entity wants the most,” then you can agree, whatever someone wants is “good,” be it paperclips or eudaemonia. On the other hand, I’m not sure we should do this, there are some hypothetical entities I can imagine where I can’t see it as ever being good that they get what they want. For instance I can imagine a Human-Torture-Maximizer that wants to do nothing but torture human beings. It seems to me that even if there were a trillion Human-Torture-Maximizers and one human in the universe it would be bad for them to get what they want.
For more neutral, but still alien preferences, I’m less sure. It seems to me that I have a right to stop Human-Torture-Maximizers from getting what they want. But would I have the right to stop paperclippers? Making the same paperclip over and over again seems like a pointless activity to me, but if the paperclippers are willing to share part of the universe with existing humans do I have a right to stop them? I don’t know, and I don’t think Eliezer does either.
I think that we, and most humans, have the same basic desires, where we differ is the object of those desires, and the priority of those desires.
For instance, most people desire romantic love. But those desires usually have different objects, I desire romantic love with my girlfriend, other people desire it with their significant others. Similarly, most people desire to consume stories, but the object of that desire differs, some people like Transformers, others The Notebook.
Similarly, people often desire the same things, but differ as to their priorities, how much of those things they want. Most people desire both socializing, and quiet solitude, but some extroverts want lots of one and less of the other, while introverts are the opposite.
In the case of the paerclippers, my first instinct is to regard opposing paperclipping as no different from the many ways humans have persecuted each other for wanting different things in the past. But then it occurred to me that paperclip-maximizing might be different because most persecutions in the past involve persecuting people who have different objects and priorities, not people who actually have different desires. For instance homosexuality is the same kind of desire as heterosexuality, just with a different object (same sex instead of opposite).
Does this mean it isn’t bad to oppose paperclipping? I don’t know, maybe, but maybe not. Maybe we should just try to avoid creating paperclippers or similar creatures so we don’t have to deal with it.
This seems like a difference in priority, rather than desire, as most people would prefer differing proportions of both both. It’s still a legitimate disagreement, but I think it’s more about finding a compromise between conflicting priorities, rather than totally different values.
Compounding this problem is the fact that people value diversity to some extent. We don’t value all types of diversity obviously, I think we’d all like to live in a world where people held unanimous views on the unacceptability of torturing innocent people. But we would like other people to be different from us in some ways. Most people, I think, would rather live in a world full of different people with different personalities than a world consisting entirely of exact duplicates (in both personality and memory) of one person. So it might be impossible to reach full agreement on those other values without screwing up the achievement of the Value of Diversity.
I’m sorry, there’s an ambiguity there—when you say “the sum of these”, are you summing across the moral values and imperatives of a single person, or of humanity as a whole?
You are quite correct. I apologise; I changed that example several times from where I started, and it seems that one of my last-minute changes actually made it a worse example (my aim was to try to show how the explicit aim of self-preservation could be a reasonable moral aim, but in the process I made it not a moral aim at all). I should watch out for that in the future.
I’ve always felt that was because one of the effects of great power, is that it’s so very easy to let everyone die. With great power, as Spiderman is told, comes great responsibility; one way to ensure that you’re not letting your own power go to your head, is by refusing to not-rescue anyone. After all, if the average science hero lets everyone he thinks is an idiot die, then who would be left?
Sometimes there’s a different reason, though; Sherlock Holmes would ignore a straightforward and safe case to catch a serial killer in order to concentrate on a tricky and deadly case involving a stolen diamond; he wasn’t in the detective business to help people, he was in the detective business in order to be challenged, and he would regularly refuse to take cases that did not challenge him.
(That’s probably a fair example as well, actually; for Holmes, only the challenge, the mental stimulation of a worthy foe, is important; for Superman, what is important is the saving of lives, whether from a mindless tsunami or Lex Luthor’s latest plot).
Hmmm. If you’re willing to accept zero, or near-zero, as a priority, then that statement can apply to any two sets of desires. Consider Sherlock holmes and a paperclipper; Holmes’ desire for mental stimulation is high-priority, his desire for paperclips is zero-priority, while the paperclipper’s desire for paperclips is high priority, and its desire for mental stimulation is zero-priority. (Some desires may have negative priority, which can then be interpreted as a priority to avoid that outcome—for example, my desire to immerse my hand in acid is negative, but a masochist may have a positive priority for that desire)
This implies that, in order to meaningfully differentiate the above statement from “some people have different desires”, I may have to designate some very low priority, below which the desire is considered absent (I may, of course, place that line at exactly zero priority). Some desires, however, may have no priority on their own, but inherit priority from another desire that they feed into; for example, a paperclipper has zero desire for self-preservation on its own, but it will desire self-preservation so that it can better create more paperclips.
Now, given a pool of potential goals, most people will pick out several desires from that pool, and there will be a large overlap between any two people (for example, most humans desire to eat—most but not all, there are certain eating disorders that can mess with that), and it is possible to pick out a set of desires that most people will have high priorities for.
It’s even probably possible to pick out a (smaller) set of desires such that those who do not have those desires at some positive priority are considered psychologically unhealthy. But such people nonetheless do exist.
In my personal view, it is neutral to paperclip or to oppose paperclipping. It becomes bad to paperclip only when the paperclipping takes resources away from something more important.
And there are circumstances (somewhat forced circumstances) where it could be good to paperclip.
There exist people who would place negative value on the idea of following the instuctions of any legitimate authority. (They tend to remain a small and marginal group, because they cannot in turn form an authority for followers to follow without rampant hypocrisy).
Yes, diversity has many benefits. The second-biggest benefit of diversity is that some people will be more correct than others, and this can be seen in the results they get; then everyone can re-diversify around the most correct group (a slow process, taking generations, as the most successful group slowly outcompetes the rest and thus passes their memes to a greater and/or more powerful proportion of the next generation). By a similar tokem, it means that one something happens that detroys one type of person, it doesn’t destroy everyone (bananas have a definite problem there, being a bit of a monoculture).
The biggest benefit is that it leads to social interaction. A completely non-diverse society would have to be a hive mind (or different experiences would slowly begin to introduce diversity), and it would be a very lonely hive mind, with no-one to talk to.
Nearly all of humanity as a whole. There are obviously some humans who don’t really value morality, we call them sociopaths, but I think most humans care about very similar moral concepts. The fact that people have somewhat different personal preferences and desires at first might seem to challenge this idea, but I don’t really think it does. It just means that there are some desires that generate the same “value” of “good” when fed into the “equation.” In fact, if diversity is a good, as we discussed previously, then people having different personal preferences might in fact be morally desirable.
That’s a good point. I was considering using the word “proportionality” instead of “priority” to better delineate that I don’t accept zero as a priority, but rejected it because it sounded clunky. Maybe I shouldn’t have.
I agree with that. What I’m wondering is, would I have a moral duty to share resources with a paperclipper if it existed, or would pretty much any of the things I spend the resources on if I kept them for myself count (i.e. eudaemonic things) as “something more important.”
I think there might actually be lots of people like this, but most appear normal because they place even greater negative value on doing something stupid because they ignored good advice just because it came from an authority. In other words, following authority is a negative terminal value, but an extremely positive instrumental value.
Exactly. I would still want the world to be full of a diverse variety of people, even if I had a nonsentient AI that was right about everything and could serve my every bodily need.
Okay then, next question; how do you decide which people to exclude? You say that you are excluding sociopaths, and I think that they should be excluded; but on exactly what basis? If you’re excluding them simply because they fail to have the same moral imperatives as the ones that you think are important, then that sounds very much like a No True Scotsman argument to me. (I exclude them mainly on an argument of appeal to authority, myself, but that also has logic problems; in either case, it’s a matter of first sketching out what the moral imperative should be, then throwing out the people who don’t match).
And for a follow-up question; is it necessary to limit it to humanity? Let us assume that, ten years from now, a flying saucer lands in the middle of Durban, and we meet a sentient alien form of life. Would it be necessary to include their moral preferences in the equation as well?
Even if they are Pebblesorters?
It may be, but only within a limited range. A serial killer is well outside that range, even if he believes that he is doing good by only killing “evil” people (for some definition of “evil”).
Hmmm. I think I’d put “buying a packet of paperclips for the paperclipper” as on the same moral footing, more or less, as “buying an icecream for a small child”. It’s nice for the person (or paperclipper) recieving the gift, and that makes it a minor moral positive by increasing happiness by a tiny fraction. But if you could otherwise spend that money on something that would save a life, then that clearly takes priority.
Hmmm. Good point; that is quite possible. (Given how many people seem to follow any reasonably persuasive authority, though, I suspect that most people have a positive priority for this goal—this is probably because, for a lot of human history, peasants who disagreed with the aristocracy tended to have fewer descendants unless they all disagreed and wiped out said aristocracy).
Here’s a tricky question—what exactly are the limits of “nonsentient”? Can a nonsentient AI fake it by, with clever use of holograms and/or humanoid robots, cause you to think that you are surrounded by a diverse variety of people even when you are not (thus supplying the non-bodily need of social interaction)? The robots would all be philosophical zombies, of course; but is there any way to tell?
I don’t think I’m coming across right. I’m not saying that morality is some sort of collective agreement of people in regards to their various preferences. I’m saying that morality is a series of concepts such as fairness, happiness, freedom etc., that these concepts are objective in the sense that it can be objectively determined how much fairness, freedom, happiness etc. there is in the world, and that the sum of these concepts can be expressed as a large equation.
People vary in their preference for morality, most people care about fairness, freedom, happiness, etc. to some extent. But there are some people who don’t care about morality at all, such as sociopaths.
Morality isn’t a preference. It isn’t the part of a person’s brain that says “This society is fair and free and happy, therefore I prefer it.” Morality is those disembodied concepts of freedom, fairness, happiness, etc. So if a person doesn’t care about those things, it doesn’t mean that freedom, fairness, happiness, etc. aren’t part of their morality. It means that person doesn’t care about morality, they care about something else.”
To use the Pebblesorter analogy again, the fact that you and I don’t care about sorting pebbles into prime-numbered heaps isn’t because we have our own concept of “primeness” that doesn’t include 2, 3, 5 and 7. It just means we don’t care about primeness.
To make another analogy, if most people preferred wearing wool clothes but one person preferred cotton, that wouldn’t mean that that person had their own version of wool, which was cotton. It means that that person doesn’t prefer wool.
Look inward, and consider why you think most people should be included. Presumably it’s because you really care a lot about being fair. But that necessarily means that you cared about fairness before you even considered what other people might think. Otherwise it wouldn’t have even occurred to you to think about what they preferred in the first place.
The fact that most humans care, to some extent, about the various facets of morality, is a very lucky thing, a planet full of sociopaths would be most unpleasant. But it isn’t relevant to the truth of morality. You’d still think torturing people was bad if all the non-sociopaths on Earth except you were killed, wouldn’t you? If, in that devastated world, you came across a sociopath torturing another sociopath or an animal, and could stop them at no risk to yourself, you’d do it, wouldn’t you?
I suspect that your intuition comes from the fact that a central part of morality is fairness, and sociopaths don’t care about fairness. Obviously being fair to the unfair is as unwise as tolerating the intolerant.
Again, I want to emphasize that morality isn’t the “preference” part, it’s the “concepts” part. But the question of the moral significance of aliens is relevant, I think it would depend on how many of the concepts that make up morality they cared about. I think that at a bare minimum they’d need fairness and sympathy.
So if the Pebblesorters that came out of that ship were horrified that we didn’t care about primality, but were willing to be fair and share the universe with us, they’d be a morally worthwhile species. But if they had no preference for fairness or any sympathy at all, and would gladly kill a billion humans to sort a few more pebbles, that would be a different story. In that case we should probably, after satisfying ourselves that all Pebblesorters were psychologically similar, start prepping a Relativistic Kill Vehicle to point at their planet if they try something.
I don’t know if I could tell, but I’d very much prefer that the AI not do that, and would consider myself to have been massively harmed if it did, even if I never found out. My preference is to actually interact with a diverse variety of people, not to merely have a series of experiences that seem like I’m doing it.
So, OK. Suppose, on this account, that you and I both care about morality to the same degree… that is, you don’t care about morality more than I do, and I don’t care about morality more than you do. (I’m not sure how we could ever know that this was the case, but just suppose hypothetically that it’s true.)
Suppose we’re faced with a situation in which there are two choices we can make. Choice A causes a system to be more fair, but less free. Choice B leaves that system unchanged. Suppose, for simplicity’s sake, that those are the only two choices available, and we both have all relevant information about the system.
On your account, will we necessarily agree on which choice to make? Or is it possible, in that situation, that you might choose A and I choose B, or vice-versa?
I think it depends on the degree of the change. If the change is very lopsided (i.e −100 freedom, +1 fairness) I think we’d both choose B.
If we assume that the degree of change is about the same (i.e. +1 fairness, −1 freedom) it would depend on how much freedom and fairness already exist. If the system is very fair, but very unfree, we’d both choose B, but if it’s very free and very unfair we’d both choose A.
However, if we are to assume that the gain in fairness and the loss in freedom are of approximately equivalent size and the current system has fairly large amounts of both freedom and fairness (which I think is what you meant) then it might be possible that we’d have a disagreement that couldn’t be resolved with pure reasoning.
This is called moral pluralism, the idea that there might be multiple moral values (such as freedom, fairness, and happiness) which are objectively correct, imperfectly commensurable with each other, and can be combined in different proportions that are of approximately equivalent objective moral value. If this is the case then your preference for one set of proportions over the other might be determined by arbitrary factors of your personality.
This is not the same as moral relativism, as these moral values are all objectively good, and any society that severely lacks one of them is objectively bad. It’s just that there are certain combinations with different proportions of values that might be both “equally good,” and personal preferences might be the “tiebreaker.” To put it in more concrete terms, a social democracy with low economic regulation and a small welfare state might be “just as good” as a social democracy with slightly higher economic regulation and a slightly larger welfare state, and people might honestly and irresolvably disagree over which one is better. However, both of those societies would definitely be objectively better than Cambodia under the Khmer Rouge, and any rational, fully informed person who cares about morality would be able to see that.
Of course, if we are both highly rational and moral, and disagreed about A vs. B, we’d both agree that fighting over them excessively would be morally worse than choosing either of them, and find some way to resolve our disagreement, even if it meant flipping a coin.
I agree with you that in sufficiently extreme cases, we would both make the same choice. Call that set of cases S1.
I think you’re saying that if the case is not that extreme, we might not make the same choice, even though we both care equally about the thing you’re using “morality” to refer to. I agree with that as well. Call that set of cases S2.
I also agree that even in S2, there’s a vast class of options that we’d both agree are worse than either of our choices (as you illustrate with the Khmer Rouge), and a vast class of options that we’d both agree are better than either of our choices, supposing that we are as you suggest rational informed people who care about the thing you’re using “morality” to refer to.
If I’m understanding you, you’re saying in S2 we are making different decisions, but our decisions are equally good. Further, you’re saying that we might not know that our decisions are equally good. I might make choice A and think choice B is wrong, and you might make choice B and think choice A is wrong. Being rational and well-informed people we’d agree that both A and B are better than the Khmer Rouge, and we might even agree that they’re both better than fighting over which one to adopt, but it might still remain true that I think B is wrong and you think A is wrong, even though neither of us thinks the other choice is as wrong as the Khmer Rouge, or fighting about it, or setting fire to the building, or various other wrong things we might choose to evaluate.
Have I followed your position so far?
Yes, I think so.
OK, good.
It follows that if a choice can go one of three ways (c1, c2, c3) and if I think c1> c2 > c3 and therefore endorse c1, and if you think c2 > c1 > c3 and therefore endorse c2, and if we’re both rational informed people who are in possession of the same set of facts about that choice and its consequences, and if we each think that the other is wrong to endorse the choice we endorse (while still agreeing that it’s better than c3), that there are (at least) two possibilities.
One possibility is that c1 and c2 are, objectively, equally good choices, but we each think the other is wrong anyway. In this case we both care about morality, even though we disagree about right action.
Another possibility is that c1 and c2 are, objectively, not equally good. For example, perhaps c1 is objectively bad, violates morality, and I endorse it only because I don’t actually care about morality. Of course, in this case I may use the label “morality” to describe what I care about, but that’s at best confusing and at worst actively deceptive, because what I really care about isn’t morality at all, but some other thing, like prime-numbered heaps or whatever.
Yes?
So, given that, I think my question is: how might I go about figuring out which possibility is the case?
I’d say it’s misleading to say we thought the other person was “wrong,” since in this context “wrong” is a word usually used to describe a situation where someone is in objective moral error. It might be better to say: “c1 and c2 are, objectively, equally morally good, but we each prefer a different one for arbitrary, non-moral reasons.”
This doesn’t change your argument in any way, I just think it’s good to have the language clear to avoid accidentally letting in any connotations that don’t belong.
This is not something I have done a lot of thinking on since the odds of ever encountering such a situation are quite low at the present. It seems to me, however, that if you are this fair to your opponent, and care this much about finding out the honest truth, that you probably care at least somewhat about morality.
(This brings up an interesting question, which is: might there be some “semi-sociopathic” humans who care about morality, but incrementally, not categorically? That is, if one of these people was rational, fully informed, lacking in self deception, and lacked akrasia, they would devote maybe 70% of their time and effort to morality and 30% to other things? Such a person, if compelled to be honest, might admit that c2 is morally worse than c1, but they don’t care because they’ve used up their 70% of moral effort for the day. It doesn’t seem totally implausible that such people might exist, but maybe I’m missing something about how moral psychology works, maybe it doesn’t work unless it’s all or nothing.)
As for determining if your opponent care about morality, you might look to see if they exhibit any of the signs of sociopathy. You might search their arguments for signs of anti-epistemology, or plain moral errors. If you don’t notice any of these things, you might assign a higher probability to prediction that your disagreement is to preferring different forms of pluralism
Of course, in real life perfectly informed, rational humans who lack self deception, akrasia, and so on do not exist. So you should probably assign a much, much, much higher probability to one of those things causing your disagreement.
OK. In which case I can also phrase my question as, when I choose c1 over c2, how can I tell whether I’m making that choice for objective moral reasons, as opposed to making that choice for arbitrary non-moral reasons?
You’re right that it doesn’t really change the argument, I’m just trying to establish some common language so we can communicate clearly.
For my own part, I agree with you that ignorance and akrasia are major influences, and I also believe that what you describe as “incremental caring about morality” is pretty common (though I would describe it as individual values differing).
Wikipedia’s page on internalism and externalism calls an entity that understands moral arguments, but is not motivated by them an “amoralist.” We could say that a person who cares about morality incrementally to have individual values that are part moralist and part amoralist.
It’s hard to tell how many people are like this due to the confounding factors of irrationality and akrasia. But I think it’s possible that there are some people who, if their irrationality and akrasia were cured, would not act perfectly moral. These people would say “I know that the world would be a better place if I acted differently, but I only care about the world to a limited extent.”
However, considering that these people would be rational and lack akrasia, they would still probably do more moral good than the average person does today.
Would they say necessarily say that? Or might they instead say “I know you think the world would be a better place if I acted differently, but actually it seems to me the world is better if I do what I’m doing?”
That depends on if they are using the term “better” to mean “morally better” or “more effectively satisfies the sum total of all my values, both moral or non-moral.”
If the person is fully rational, lacking in self-deception, is being totally honest with you, and you and they had both agreed ahead of time that the word “better” means “morally better” then yes, I think they would say that. If they were lying, or they thought you were using the term “better” to mean “more effectively satisfies the sum total of all my values, both moral or non-moral,” then they might not.
I agree with you that IF (some of) my values are not moral and I choose to maximally implement my values, THEN I’m choosing not to act so as to make the world a morally better place, and IF I somehow knew that those values were not moral, THEN I would say as much if asked (supposing I was aware of my values and I was honest and so forth).
But on your account I still don’t see any way for me to ever know which of my values are moral and which ones aren’t, no matter how self-aware, rational, or lacking in self-deception I might be.
Also, even if I did somehow know that, and were honest and so forth, I don’t think I would say “I only care about the world to a limited extent.” By maximizing my values as implemented in the world, I would be increasing the value of the world, which is one way to express caring about the world. Rather, I would say “I only care about moral betterness to a limited extent; there are more valuable things for the world to be than morally better.”
How does a Pebblesorter know it’s piles are prime? The less intelligent and rational probably use some sort of vague intuition. The more intelligent and rational probably try dividing the numbers of pebbles by a number other than one.
If you had full knowledge of the concept of “morality” and all the various sub-concepts it included, you could translate that concept into a mathematical equation (the one I’ve been discussing with CC lately), and see if the various values of yours that you feed into it return positive numbers.
If your knowledge is more crude (i.e. if you’re a real person who actually exists), then a possible way to do it would be to divide the nebulous super-concept of “morality” into a series of more concrete and clearly defined sub-concepts that compose it (i.e. freedom, happiness, fairness, etc.). It might also be helpful to make a list of sub-concepts that are definitely not part of morality (possible candidates include malice, sadism, anhedonia, and xenophobia).
After doing that you could, if you are not self-deceived, use introspection to figure what you value. If you find that your values include the various moral sub-concepts, then it seems like you value morality. If you find yourself not valuing the moral sub-concepts, or valuing some nonmoral concept, then you do not value morality, or value non-moral things.
As Eliezer puts it:
The moral equation we are looking for isn’t something that will provide us with a ghostly essence. It is something that will allow us to sum up and aggregate all the seperate good things like truth, happiness, and sentient life, so that we can effectively determine how good various combinations of these things are relative to each other, and reach an optimal combo.
Do you want people to be happy, free, be treated fairly, etc? Then you value morality to some extent. Do you love torturing people just for the hell of it, or want to convert all the matter in the universe into paperclips? Then you, at the very least, definitely value other things than morality.
By “caring” I meant “caring about whether the world is a good and moral place.” If you instead use the word “caring” to mean “have values that assign different levels of desirability to various possible states that the world could be in” then you are indeed correct that you would not say you didn’t care about the world.
If by “valuable” you mean “has more of the things that I care about,” then yes, you could say that. Remember, however, that in that case what is “valuable” is subjective, it changes from person to person depending on their individual utility functions. What is “morally valuable,” by contrast, is objective. Anyone regardless of their utility function, can agree on whether or not the world has great quantities of things like truth, freedom, happiness, and sentient life. What determines the moral character of a person is how much they value those particular things.
Also, as an aside, when I mentioned concepts that probably aren’t part of morality earlier, I did not mean to say that pursuit of those concepts always neccessarily leads to immoral results. For instance, imagine a malicious sadist who wants to break someone’s knees. This person assaults someone else out of pure malice and breaks their knees. The injured person turns out to be an escaped serial killer who was about to kill again, and the police are able to apprehend them in their injured state. In this case the malicious person has done good. However, this is not because they have intentionally increased the amount of malicious torture in the universe. It is because they accidentally decreased the amount of murders in the universe.
I 100% agree that there is no ghostly essence of goodness.
I agree that pursuing amoral, or even immoral, values can still lead to moral results. (And also vice-versa.)
I agree that if I somehow knew what was moral and what wasn’t, then I would have a basis for formally distinguishing my moral values from my non-moral values even when my intuitions failed. I could even, in principle, build an automated mechanism for judging things as moral or non-moral. (Similarly, if a Pebblesorter knew that primeness was what it valued and knew how to factor large numbers, it would have a basis for formally distinguishing piles it valued from piles it didn’t value even when its intuitive judgments failed, and it could build an automated mechanism for distinguishing such piles.)
I agree with you that real people who actually exist can’t do this, at least not in detail.
You suggest we can divide morality into subconcepts that comprise it (freedom, happiness, fairness, etc.) and that it excludes (anhedonia, etc.). What I continue to not get is, on your account, how I do that in such a way as to ensure that what I end up with is the objectively correct list of moral values, which on your account exists, rather than some different list of values.
That is, suppose Sam and George both go through this exercise, and one of them ends up with “freedom” on their list but not “cooperation”, and the other ends up with “cooperation” but not “freedom.” On your account it seems clear that at least one of them is wrong, because the correct list of moral values is objective.
So, OK… what would we expect to experience if Sam were right? How does that differ from what we would expect to experience if George were right, or if neither of them were?
Again: how do we know that? What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does? What should I be looking for, to notice if this is true, or confirm that it isn’t? I would still want people to be happy, free, be treated fairly, etc. in either case, after all. What differences would I experience between the two cases?
Yes, that’s more or less what I mean by “caring”. More precisely I would say that caring about X consists of desiring states of the world with more X more than states of the world with less X, all else being equal, but that’s close enough to what you said.
Yes, that’s what I mean by “valuable.” And yes, absolutely, what is valuable changes from person to person. If I act to maximize my values and you act to maximize yours we might act in opposition (or we might not, depending, but it’s possible).
And I get that you want to say that if we both gave up maximizing our values and instead agreed to implement moral values, then we would be cooperating instead, and the world would be better (even if it turned out that both of us found it less valuable). What I’m asking you is how (even in principle) we could ever reach that point.
To say that a little differently: you value some things (Vg) and I value some things (Vd). Supposing we are both perfectly rational and honest and etc., we can both know what Vg and Vd are, and what events in the world would maximize each. We can agree to cooperate on maximizing the intersection of (Vg,Vd), and we can work out some pragmatic compromise about the non-overlapping stuff. So far so good; I see how we could in principle reach that point, even if in practice we aren’t rational or self-aware or honest enough to do it.
But I don’t see how we could ever say “There’s this other list, Vm, of moral values; let’s ignore Vg and Vd altogether and instead implement Vm!” because I don’t see how we could ever know what Vm was, even in principle. If we happened to agree on some list Vm, either by coincidence or due to social conditioning or for other reasons, we could agree to implement Vm… which might or might not make the world better, depending on whether Vm happened to be the objectively correct list of moral values. But I don’t see how we could ever, even in principle, confirm or deny this, or correct it if we somehow came to know we had the objectively wrong list.
And if we can’t know or confirm or deny or correct it, even in principle, then I don’t see what is added by discussing it. It seems to me I can just as usefully say, in this case, “I value happiness, freedom, fairness, etc. I will act to maximize those values, and I endorse acting this way,” and nothing is added by saying “Those values comprise morality” except that I’ve asserted a privileged social status for my values.
Well, I am basically asserting that morality is some sort of objective equation, or “abstract idealized dynamic,” as Eliezer calls it, concerned with people’s wellbeing. And I am further asserting that most human beings care very much about this concept. I think this would make the following predictions:
In a situation where a given group of humans had similar levels of empirical knowledge and a similar sanity waterline there would be far more moral agreement among them than would be predicted by chance, and far less moral disagreement than is mentally possible.
It is physically possible to persuade people to change their moral values by reasoned argument.
Inhabitants of a society who are unusually rational and intelligent will be the first people in that society to make moral progress, as they will be better at extrapolating answers out of the “equation.”
If one attempted to convert the moral computations people make into an abstract, idealized process, and determine it’s results, many people would find those results at least somewhat persuasive, and may find their ethical views changed by observing them.
All of these predictions appear to be true:
Human societies tend to have a rather high level of moral agreement between their members. Conformity is not necessarily an indication of rightness, it seems fairly obvious that whole societies have held gravely mistaken moral views, such as those that believed slavery was good. However, it is interesting that all those people in those societies were mistaken in exactly the same way. That seems like evidence that they were all reasoning towards similar conclusions, and the mistakes they made were caused by common environmental factors that impacted all of them. There are other theories that explain this data of course, (peer pressure, for instance), but I still find it striking.
I’ve had moral arguments made by other people change my mind, and changed the minds of other people by moral argument. I’m sure you have also had this experience.
It is well known that intellectuals tend to develop and adopt new moral theories before the general populace does. Common examples of intellectuals whose moral concepts have disseminated into the general populace include John Locke, Jeremy Bentham, and William Lloyd Garrison. Many of these peoples’ principles have since been adopted into the public consciousness.
Ethical theorists who have attempted to derive new ethical principles by working from an abstract, idealized form of ethics have often been very persuasive. To name just one example, Peter Singer ended up turning thousands of people into vegetarians with moral arguments that started on a fairly abstract level.
Asserting that those values comprise morality seems to be effective because it seems to most people that those values are related in some way, because they form the superconcept “morality.” Morality is a useful catchall term for certain types of values, and it would be a shame to lose it.
Still, I suppose that asserting “I value happiness, freedom, fairness, etc” is similar enough to saying “I care about morality” that I really can’t object terribly strongly if that’s what you’d prefer to do.
Why does doing that bother you? Presumably, because you care about the moral concept of fairness, and don’t want to claim an unfair level of status for you and your views. But does it really make sense to say “I care about fairness, but I want to be fair to other people who don’t care about it, so I’ll go ahead and let them treat people unfairly, in order to be fair.” That sounds silly, doesn’t it? it has the same problems that come with being tolerant of intolerant people.
All of those predictions seem equally likely to me whether Sam is right or George is, so don’t really engage with my question at all. At this point, after several trips ’round the mulberry bush, I conclude that this is not because I’m being unclear with my question but rather because you’re choosing not to answer it, so I will stop trying to clarify the question further.
If I map your predictions and observations to the closest analogues that make any sense to me at all, I basically agree with them.
It is.
It doesn’t bother me; it’s a fine thing to do under some circumstances. If we can agree that that’s what we’re doing when we talk about “objective morality,” great. If not (which I find more likely), never mind.
As above, I don’t see what the word “moral” is adding to this sentence. But sure, unfairly claiming status bothers me to the extent that I care about fairness. (That said, I don’t think claiming status by describing my values as “moral” is unfair; pretty much everybody has an equal ability to do it, and indeed they do. I just think it confuses any honest attempt at understanding what’s really going on when we decide on what to do.)
It depends on why and how I value (“care about”) fairness.
If I value it instrumentally (which I do), then it makes perfect sense to say that being fair to people who treat others unfairly is net-valuable, although it might be true or false in any given situation depending on what is achieved by the various kind of fairness that exist in tension in that situation.
Similarly, if I value it in proportion to how much of it there is (which I do), then it makes sense to say that, although it might be true or false depending on how much fairness is gained or lost by doing so.
(nods) Totally. And the ability to phrase ideas in silly-sounding ways is valuable for rhetorical purposes, although it isn’t worth much as an analytical tool.
I’m really sorry, I was trying to kill two birds with one stone and simultaneously engage that question and your later question [“What would I expect to experience differently if, instead, happiness, freedom, fairness, etc. turned out not to be aspects of morality, just like maximizing paperclips does?”] at the same time, and I ended up doing a crappy job of answering both of them. I’ll try to just answer the Sam and George question now.
I’ll start by examining the Pebblesorters P-George and P-Sam. P-George thinks 9 is p-right and 16 is p-wrong. P-Sam thinks 9 is p-wrong and 16 is p-right. They both think they are using the word “p-right” to refer to the same abstract, idealized process. What can they do to see which one is right?
They assume that most other Pebblesorters care about the same abstract process they do, so they can try to persuade them and see how successful they are. Of course, even if all the Pebblesorters agree with one of them, that doesn’t necessarily mean that one is p-correct, those sorters may be making the same mistake as P-George or P-Sam. But I think it’s non-zero Bayesian evidence of the p-rightness of their views.
They can try to control for environmentally caused error by seeing if they can also persuade Pebblesorters who live in different environments and cultures.
They can find the most rational and p-sane Pebblesorting societies and see if they have an easier time persuading them.
They can actually try to extrapolate what the abstract, idealized equation that the word “p-right” represents is and compare it to their views. They read up on Pebblesorter philospher’s theories of p-rightness and see how they correlate with their views. Pebblesorting is much simpler than morality, so we know that the abstract, idealized dynamic that the concept “p-right” represents is “primality.” So we know that P-Sam and P-George are both partly right and partly wrong, 9 and 16 both aren’t prime.
Now let’s translate that into human.
We would expect if Sam was right and George was wrong:
He would have an easier time persuading non-sociopathic humans of the rightness of his views than George, because his views are closer to the results of the equation those people have in their head.
If he went around to different societies with different moral views and attempted to persuade the people there of his views he should, on average, also have an easier time of it than George, again because his views are closer to the results of the equation those people have in their head.
Societies with higher levels of sanity and rationality should be especially easily persuaded, because they are better at determining what the results of that equation would be.
When Sam compared his and George’s views to views generated by various attempts by philosophers to create an abstract idealized version of the equation (ie. moral theories), his view should be a better match to many of them, and the results they generate, than George’s are.
The problem is that the concept of morality is far more complex than the concept of primality, so finding the right abstract idealized equation is harder for humans than it is for Pebblesorters. We still haven’t managed to do it yet. But I think that comparing Sam and George’s views to the best approximations we have so far (various forms of consequentialism, in my view) we can get some Bayesian evidence of the rightness of their views.
If George is right, he will achieve these results instead of Sam. If they are both wrong, they will both fail at doing these things.
Sorry, I was probably being unclear as to what I meant because I was trying to sound clever. When I said it was silly to be fair to unfair people what I meant was that you should not regard their advice on how to best treat other people with the same consideration you’d give to a fair-minded person’s advice.
For instance, you wouldn’t say “I think it’s wrong to enslave black people, but that guy over there thinks it’s right, so let’s compromise and believe it’s okay to enslave them 50% of the time.” I suppose you might pretend to believe that if the other guy had a gun and you didn’t, but you wouldn’t let his beliefs affect yours.
I did not mean that, for example, if you, two fair-minded people, and one unfair-minded person are lost in the woods and find a pie, that you shouldn’t give the unfair-minded person a quarter of the pie to eat. That is an instance where it does make sense to treat unfair people fairly.
OK. Thanks for engaging with the question; that was very helpful. I now have a much better understanding of what you believe the differences-in-practice between moral and non-moral values are.
Just to echo back what I’m hearing you say: to the extent that some set of values Vm is easier to convince humans to adopt than other sets of values and easier to convince sane, rational societies to adopt than less sane, less rational societies and better approximates the moral theories created by philosophers than other sets of values, to that extent we can be confident that Vm is the set of values that comprise morality.
Did I get that right?
Regarding fairmindedness: I endorse giving someone’s advice consideration to the extent that I’m confident that considering their advice will implement my values. And, sure, it’s unlikely that the advice of an unfairminded person would, if considered, implement the value of fairness.
Yes, all those things provide small bits of Bayesian evidence that Vm is closer to morality than some other set of values.
It seems very likely that a person who cares a certain amount about morality, and a certain amount about money, would be willing to compromise his morality if given sufficient money. Such a mental model would form the basis of bribery. (It doesn’t have to be money, either, but the principle remains the same).
So a semi-sociopathic person would be anyone who could be bribed into completely disregarding morality.
On this account, we could presumably also categorize a semi-semi-sociopathic person as one who could be bribed into partially disregarding the thing we’re labeling “morality”. And of course bribes needn’t be money… people can be bribed by all kinds of things. Social status. Sex. Pleasant experiences. The promise of any or all of those things in the future.
Which is to say, we could categorize a semi-semi-sociopath as someone who cares about some stuff, and makes choices consistent with maximizing the stuff they care about, where some of that stuff is what we’re labeling “morality” and some of it isn’t.
We could also replace the term “semi-semi-sociopath” with the easier to pronounce and roughly equivalent term “person”.
It’s also worth noting that there probably exists stuff that we would label “morality” in one context and “bribe” in another, were we inclined to use such labels.
Ah, I think I see your point. What you’re saying—and correct me if I’m wrong—is that there is some objective True Morality, some complex equation that, if applied to any possible situation, will tell you how moral a given act is.
This is probably true.
This equation isn’t written into the human psyche; it exists independantly of what people think about morality. It just is. And even if we don’t know exactly what the equation is, even if we can’t work out the morality of a given act down to the tenth decimal place, we can still apply basic heuristics and arrive at a usable estimate in most situations.
My question is, then—assuming the above is true, how do we find that equation? Does there exist some objective method whereby you, I, a Pebblesorter, and a Paperclipper can all independently arrive at the same definition for what is moral (given that the Pebblesorter and Paperclipper will almost certainly promptly ignore the result)?
(I had thought that you were proposing that we find that equation by summing across the moral values and imperatives of humanity as a whole—excluding the psychopaths. This is why I asked about the exclusion, because it sounded a lot like writing down what you wanted at the end of the page and then going back and discarding the steps that wouldn’t lead there; that is also why I asked about the aliens).
Yes, I think we’re in agreement on that. (Though this does suggest that ‘sentient’ may need a proper definition at some point).
In the same way as there exists a True Set of Prime Numbers, and True Measure of How Many Paperclips There Are...
Even though the equation exists independently of our thoughts (the same way primality exists independently from Pebblesorter thoughts) fact that we are capable of caring about the results given by the equation means we must have some parts of it “written” in our heads, the same way Pebblesorters must have some concept of primality “written” in their heads. Otherwise, how would we be capable of caring about its results?
I think that probably evolution metaphorically “wrote” a desire to care about the equation in our heads because if humans care about what is good and right it makes it easier for them to cooperate and trust each other, which has obvious fitness advantages. Of course, the fact that evolution did a good thing by causing us to care about morality doesn’t mean that evolution is always good, or that evolutionary fitness is a moral justification for anything. Evolution is an amoral force causes many horrible things to happen. It just happened that in this particular instance, evolution’s amoral metaphorical “desires” happened to coincide with what was morally good. That coincidence is far from the norm, in fact, evolution probably deleted morality from the brains of sociopaths because double-crossing morally good people also sometimes confers a fitness advantage.
So how do we learn more about this moral equation that we care about? One common form of attempting to get approximations of it in philosophy is called reflective equilibrium, where you take your moral imperatives and heuristics and attempt to find the commonalities and consistencies they have with each other. It’s far from perfect, but I think that this method has produced useful results in the past.
Eliezer has proposed what is essentially a souped up version of reflective equilibrium called Coherent Extrapolated Volition. He has argued, however, that the primary use of CEV is in designing AIs that won’t want to kill us, and that attempting to extrapolate other people’s volition is open to corruption, as we could easily fall to the temptation to extrapolate it to something that personally benefits us.
Again, we could probably get closer through reflective equilibrium, and by critiquing the methods and results of each other’s reflections. If you somehow managed to get a Pebblesorter or a Paperclipper to do it too, they might generate similar results, although since they don’t intrinsically care about the equation you would probably have to give them some basic instructions before they started working on the problem.
If we assume that most humans care about acting morally, doing research about what people’s moral imperatives are might be somewhat helpful, since it would allow us to harvest the fruits of other people’s moral reflections and compare them with our own. We can exclude sociopaths because there is ample evidence that they care nothing for morality.
Although I suppose, that a super-genius sociopath who had the basic concept explained to them might be able to do some useful work in the same fashion that a Pebblesorter or Paperclipper might be able to. Of course, the genius sociopath wouldn’t care about the results, and probably would have to be paid a large sum to even agree to work on the problem.
Hmmm. That which evolution has “written” into the human psyche could, in theory, and given sufficient research, be read out again (and will almost certainly not be constant across most of humanity, but will rather exist with variations). But I doubt that morality is all in out genetic nature; I suspect that most of it is learned, from our parents, aunts, uncles, grandparents and other older relatives; I think, in short, that morality is memetic rather than genetic. Though evolution still happens in memetic systems just as well as in genetic systems.
Hmmm. Looking at the wikipedia article, I can expect reflective equilibrium to produce a consistent moral framework. I also expect a correct moral framework to be consistent; but not all consistent moral frameworks are correct. (A paperclipper does not have what I’d consider a correct moral framework, but it does have a consistent one).
If you start out close to a correct moral framework, then reflective equilibrium can move you closer, but it doesnt necessarily do so.
Hmmm. The primary use of trying to find the True Morality Equation, to my mind, is to work it into a future AI. If we can find such an equation, prove it correct, and make an AI that maximises its output value, then that would be an optimally moral AI. This may or may not count as Friendly, but it’s certainly a potential contender for the title of Friendly.
Carrying through this method to completion could give us—or anyone else—an equation. But is there any way to be sure that it necessarily gives us the correct equation? (A pebblesorter may actually be a very good help in resolving this question; he does not care about morality, and therefore does not have any emotional investment in the research).
The first thought that comes to my mind, is to have a very large group of researchers, divide them into N groups, and have each of these groups attempt, independently, to find an equation; if all of the groups find the same equation, this would be evidence that the equation found is correct (with stronger evidence at larger values of N). However, I anticipate that the acquired results would be N subtly different, but similar, equations.
That’s possible. But memetics can’t build morality out of nothing. At the very least, evolved genetics has to provide a “foundation,” a part of the brain that moral memes can latch onto. Sociopaths lack that foundation, although the research is inconclusive as to what extent this is caused by genetics, and what extent it is caused by later developmental factors (it appears to be a mix of some sort).
Yes, that’s why I consider reflective equilibrium to be far from perfect. Depending on how many errors you latch onto, it might worsen your moral state.
Considering how morally messed up the world is now, even an imperfect equation would likely be better (closer to being correct) than our current slapdash moral heuristics. At this point we haven’t even achieved “good enough,” so I don’t think we should worry too much about being “perfect.”
That’s not inconceivable. But I think that each of the subtly different equations would likely be morally better than pretty much every approximation we currently have.
That sounds plausible, yes.
Hmmm. Finding an approximation to the equation will probably be easier than step two; encouraging people worldwide to accept the approximation. (Especially since many people who do accept it will then promptly begin looking for loopholes; either to use or to patch them).
However, if the correct equation cannot be found, then this means that the Morality Maximiser AI cannot be designed.
That’s true, what I was trying to say is that a world ruled by a 99.99% Approximation of Morality Maximizer AI might well be far far better than our current one, even if it is imperfect.
Of course, it might be a problem if we put the 99.99% Approximation of Morality Maximizer AI in power, then find the correct equation, only to discover that the 99AMMAI is unwilling to step down in favor of the Morality Maximizer AI. On the other hand, putting the 99AMM AI in power might be the only way to ensure a Paperclipper doesn’t ascend to power before we find the correct equation and design the MMAI. I’m not sure whether we should risk it or not.
Well, if we’re really going to take Omelas seriously as our test case, then presumably we also have to look at how much that “extra happiness” (or whatever else we’re putting in the plus column) is reduced by those who walk away from it, and by those who are traumatized by it, and so forth. It might turn out that increasing the population doesn’t help.
But that’s just a quibble. I basically agree: once we swallow the assumption that for some reason we neither understand nor can ameliorate, the happiness of the many ineluctably depends on the misery of the few, then a total-utilitarian approach either says that equality is the most important factor in utility (which is a problem like you describe), or endorses the few being miserable.
That’s quite an assumption to swallow, though. I have no reason to believe it’s true of the world I live in.
A weaker version that might be true of the world I actually live in is that concentrating utility-generating resources in fewer hands results in higher total utility-from-all-sources-other-than-equality (Ua) but more total-disutility-from-inequality (Ub). But it’s not quite as clear that our (Ua, Ub) preferences are lexicographic.
Doubling the population should double the happiness; double the trauma; double the people who walk away. The end result should be (assuming a high enough population that the Law of Large Numbers is a reasonable heuristic) about twice the utility.
Consider the case of farmland; larger farms produce more food-per-acre than smaller farms. (Why? Because larger farms attract commercial farmers with high-intensity farming techniques; and they can buy better farming equipment with their higher profits). Now, in the case of farmland, the optimal scenario is not equality; you don’t want everyone to have the same amount of farmland, you want those who are good at farming to have most of it. (For a rather dramatic example of this, see the Zimbabwe farm invasions).
On the other hand, consider the case of food itself. Here, equality is a lot more important; giving one man food for a hundred while ninety-nine men starve is clearly a failure case, as a lot of food ends up going rotten and ninety-nine people end up dead.
So the optimal (Ua, Ub) ordering depends on exactly what it is that is being ordered; there is no universally correct ordering.
You seem to be assuming a form of utility that is linear with happiness, with trauma, with food-per-acre, with starving people, etc.
I agree with you that if we calculate utility this way, what you say follows.
It’s not clear to me that we ought to calculate utility this way.
Hmmm. There are other ways to calculate utility, yes, and some of them are very likely better than linear. But all of them should at least be monotonically increasing with increased happiness, lower trauma, etc. There isn’t some point of global happiness where you can say that global happiness above this level is worse than global happiness at this level, if all else remains constant. The increase may be smaller for higher starting levels of happiness, but it should be an increase.
Such a system can either be bounded above by a maximum value, which it approaches asymptotically (such that no amount of global happiness alone can ever be worth, say, ten billion utilions, but it can approach arbitrarily close to that amount), or it can be unbounded (in which case enough global happiness can counter any finite amount of negative effects). A linear system would be unbounded, and my comments above can be trivially changed to fit with any unbounded system (but not necessarily with a bounded system).
It’s not clear to me whether it should be a bounded or an unbounded system..
OK, so we agree that doubling the population doesn’t provide twice the utility, but you’re now arguing that it at least increases the utility (at least, up to a possible upper bound which might or might not exist).
This depends on the assumption that the utility-increasing aspects of increased population increase with population faster than the utility-decreasing aspects of increased population do. Which they might not.
...you know, it’s only after I read this comment that I realised that you’re suggesting that the utility-decreasing aspects may not use the same function as the utility-decreasing aspects. That is, what I was doing was mathematically equivalent to first linearly combining the separate aspects, and only then feeding that single number to a monotonically increasing nonlinear function.
Now I feel somewhat silly.
But yes, now I see that you are right. There are possible ethical models (example: bounded asymptotic increase for positive utility, unbounded linear decrease for negative utility) wherein a larger Omelas could be worse than a smaller Omelas, above some critical maximum size. In fact, there are some functions wherein an Omelas of size X could have positive utility, while an Omelas of size Y (with Y>X) could have negative utility.
Yup. Sorry I wasn’t clearer earlier; glad we’ve converged.
If you’re going to do that, why not just directly use happiness and fulfillment?
I cannot create an entire ethical framework, for everyone to follow, on any basis, and expect that it will be able to hold up for the next thousand years. If I try, I will fail, and this is why: because people cheat. Many intelligent agents will poke at the rules, seeking a possible exploit thereof that enhances their success at the possible expense of their neighbours’ success. Over the next thousand years, there will be thousands, probably millions, of such intelligent agents hunting for, and attempting to exploit, flaws in the system; people who stick by the letter of the rule, and avoid the spirit of the rule. I cannot create an entire ethical framework, because I cannot outwit thousands or millions of future peoples’ attempts to find and exploit gaps and loopholes in my framework.
Hence, the best that I can do is to find a system that has already endured a period of field testing and that hasn’t broken yet; and perhaps attempt a small, incremental improvement (no more) in order to test that improvement.
What does that have to do with the situation at hand? Morality is an abstract division of actions into right and wrong, not some set of laws laid down by philosophers on the rest of the population. If you’re trying to work out what you mean by “morality” and use some criteria (such as something including happiness and fulfillment of populations which adopt that definition) to choose from a bunch of alternatives, then probably those criteria themselves are the most accurate definition of “morality” you could hope to find. I might add, in [almost] exactly the same way that a program which writes and then executes a program to add two numbers is, in fact, itself a program that adds two numbers.
You can write out your final definition in legalese later, if the situation calls for it.
Morality comes with an implicit rule; when it says that “this action is the right action to take in this situation”, then the implicit rule is “if you find yourself in this situation, take this action”. There is usually no Morality Policeman ready to administer punishment if the rule is not followed, and the choice to follow the rule or not remains; but the rule is there.
The difficulty is that I know that the algorithm that I am following is very likely not to fulfil the criteria in the very best possible way; merely in (more or less) the best possible way that they have been fulfilled in the past. If I simply list the criteria, then I falsely imply that the chosen system of morality is the best fit for those criteria; and I am trying to avoid that implication.
Define it, or defend it? There are a lot of defences, but not so much definitions.
I think the metaphor misses something important here, because the number of pebbles seems completely arbitrary. What, if anything, would change if in the pebble-sorters’ ancestral environment, preferring 13-pebble heaps was adaptive, but preferring 11-pebble heaps (or spending resources on that that do) was not?
Preferring other people like Larry to be homosexual is adaptive for me. And it is the judgement by others (and the implicit avoidance of that through shame) that we are considering here. That said:
Absolutely, and the entire line of reasoning relies on conveying the speaker’s own morality (“it is second-order ‘right’ to be homosexual”) on others without making it explicit.
The same reason sorting pebbles into correct heaps was adaptive in the first place.
EDIT: Wait, does it matter that homosexuality is probably not adaptive?
That was the point of my comment. There is a large disanalogy between heterosexuality and 13-pebble heap preference (namely, the first highly adaptive, but the second has no apparent reason to be). Although, I’m not sure if that is enough to break the metaphor.
There are many properties homosexuality has but 11-pebble heap preference don’t, and vice versa. Why is evolutionary maladaptiveness worth pointing out, is my question.
Well, if moral norms are the Nash equilibria that result from actual historical bargaining situations (that are determined largely by human nature and the ancestral environment), then it seems somewhat relevant. If moral norms are actually imperative sentences uttered by God, then it seems completely irrelevant. Etc...
I suppose whether or not the pebble-sorting metaphor is good depends on which meta-ethical theory is true. In other words, I’m agreeing with PhilGoetz; Example 2 and Example 3 are only in separate classes of meta-wants assuming a (far from universally shared) moral system.
I would regard moral norms as useful heuristics for achieving morally good results, not as morality in and of itself.
I think that some sort of ethical naturalism (or “moral cognitivism” as Eliezer calls it) is correct, where “morally good” is somewhat synonymous with “helps people live lives full of positive values like love, joy, freedom, fairness, high challenge, etc.” There is still much I’m not sure of, but I think, that is probably pretty close to the meaning of right. In Larry’s case I would argue that homosexual relationships usually do help people live such lives.
Oh, you mean that humans might genuinely dislike homosexuality as a terminal value, because evo-psych.
… huh.
Incidentally, it’s easier to sort pebbles into heaps of 11. The original pebblesorters valued larger heaps, but had a harder time determining their correctness.
That’s why I was careful to refer to them as 11-Pebble and 13-Pebble Favorers. They do value other sizes of pebble heaps, 11 and 13 are just the numbers they do most frequently. Or perhaps 11 and 13 are the heaps they like making in their personal time, but they like larger prime numbers for social pebble-sorting endeavors. The point is, I said they “favored” that size because I wanted to make sure that the ease of sorting the piles didn’t seem too relevant, since that would distract away from the central metaphor.
Oops.
I suspect we’re doing some extrapolation here in order to distinguish these cases. I expect that if Mimi knew more about herself and the world, and thought more clearly, she would still want to not want heroin; while I expect that if Larry knew more about himself and the world, and thought more clearly, he would be likely to reject the system of belief that causes him to think homosexuality immoral.
Alternatively, after sobering up, Mimi might decide that experiencing heroine high makes her life so much more fulfilling, that the much shortened life expectancy of a heroine addict doesn’t seems to be a fair price to pay for it.
As usual it is all up to personal definition of utility.
Contra Cyan & Alicorn, I am inclined to go with PhilGoetz and “punt it off to your moral system, or your expected-value computations.”
Trying to change your homosexual desires will probably fail and create a lot of collatoral damage. I would guess that trying to change your desire for heroin is somewhat more likely to succeed, though I’m willing to consider the argument that heroin addicts should accept their addiction but attempt to minimize its harmful side effects.
I think the distinction is that we think of Mimi as wishing to revoke a decision made of her own free will; not so with Larry.
Mimi grew up in a society that taught her that heroin is bad. Larry grew up in a society that taught him that homosexuality is bad. How do you tell the difference from their point of view?
I think it probably has something to do with the fact that Mimi (probably) wasn’t born addicted to heroin (and even if she was, we can point to the behavior that caused it), whereas the consensus seems to be that homosexuality is innate.
There must be more to it than that. If Larry were innately born attracted to children rather than to men, we probably wouldn’t say it was okay.
It’s not about whether it’s okay, it’s about whether it’s “part of who he is” or an alien intrusion.
Okay.
I’m Paul Erdos. I’ve been taking amphetamine and ritalin for 20-odd years to enhance my cognitive performance. In general I want to want these drugs, because they help me do good, important and enjoyable work, which is impossible for me without them.
I can stop wanting these drugs when I want to, like when my friend bet me $500 that I couldn’t. I wanted to win that bet, so I wanted not to want the drugs, so I stopped wanting them. Was that my only motivation?
Also, I don’t want others to want to want amphetamines just because I want to want amphetamines.
A while ago I took Euler’s place as the most prolific mathematician of all time.
Paul Erdös did it regularly, yes. Successfully, it seems — but I wonder about the costs. Does anyone have consistent data on that?
Picking only Erdös’ case, would, I’m afraid, be a case of both survivorship bias and hasty generalization.
That doesn’t solve PhilGoetz’s example though. And in the original version of Larry, his parents might very well say that his revulsion at homosexual acts is “who he is” and his sexual feelings the “alien intrusion”. Are these concepts anything but a way of making disguised moral judgements? Is “who someone really is” just “who I would prefer them to be”?
Then again, another attitude to Larry is that his sexual feelings are who he really is, but that resisting them is a cross he has to bear. (I believe this is the Roman Catholic view.) So I don’t think the concept of authenticity solves these problems.