I actually think that there is a fundamental problem with initial question that makes the answer different from the one Eliezer thinks.
If “shut up and multiply’ was always the right answer, if you could break everything down to utilitons and go with whatever provides the most utility for the most people, then the logical thing to do would be to create a trillion intelligent beings in the solar system who do nothing but enjoy pleasure all the time. But he’s made it clear that he doesn’t believe that, that he believes that humans have a wide variety of values that are irreducible, that are fundamentally different, and that if you give up some of those values to satisfy others, you lose something important, and we all end up worse off. When it comes to fundamentally different values, like “freedom”, that you can’t just say “X freedom is worth Y pleasure, so if you can create Y+1 pleasure you should give up on the concept of freedom”, not for any value of X and Y.
In this case, I would argue that the value “nobody should suffer from torture if it can be avoided” is a fundamentally different value from “it’s nice to avoid minor annoyances when possible”, and that you can’t simply break them down into utilitons and “do the math” like he suggests.
You’re conflating “utility” and “pleasure”. The aggregation function to use for multiple people is a question of some concern, but I think most of us would agree that maximizing utility is more complex than just wireheading.
Is there a positive utility value to pleasure? Obviously there is. But can you just “shut up and multiply” that, figure out that you get more utility from wireheading then from anything else, that and then decide that that’s the right choice? Obviously not, because human value is complex, and you can’t just give up one value to create utility somewhere else, not even for arbitrarily large amounts of pleasure.
I am arguing that the same is true for things with negative utility. Is there a negative utility value to a minor annoyance? Sure. But that doesn’t mean that we should give up other values, like “torturing people is wrong” in order to avoid annoyances, not even for arbitrarily large values of annoyances.
It seems to me we make value tradeoffs all the time, based on how much we value different things at that time.
I value living. I also value happiness. Given a choice between a shorter, happier life and a longer, less-happy life, I would choose one or the other. Either way, I’m trading one value off for another. You seem to be saying that I can’t do that. I don’t see how that claim even makes sense. If I can’t choose one or the other, what can I do?
Now, if you want to say that you happen to value avoiding torture in such a way that you won’t trade any amount of minor-annoyance-avoiding for any amount of torture… that your preferences are lexicographic in this regard… well, then yes, your values are significantly different from what Eliezer is talking about, and you (specifically you) can’t “do the math”.
But it sounds like you want to say that everyone is like that. What makes you think that?
I think that most people’s values are set up like that, which is why most people have such a strong negative instinctive reaction to the whole “dust motes vs torture” question. Perhaps not, though; I obviously can’t speak for everyone.
If you had to choose between being tortured horribly for the next 50 years, or having a minor annoyance happen to you once a year for the next (insert arbitrarily large number here) years, which would you pick?
Are you so sure about that? If the number is large enough, it’s easily conceivable that at some point in your effectively infinite lifespan, you will come across a situation where that minor annoyance changes what might have been a perfectly good 50 years into 50 years of hell. So either you’ve wound up with mild scale insensitivity, a significant discount rate, or a rejection of additivity of negends.
I’m not saying I’d be better off having picked it. For the vast majority of numbers, I absolutely would not be. [EDIT: Well, assuming no knock-on effects from the torture, which EY’s initial formulation assumed.]
I’m saying it’s probably what I would, in fact, pick, if I were somehow in the epistemic state of being offered that choice. Yes, scale insensitivity and discounting play a role here, as does my confidence that I’m actually being offered an arbitrarily large number of annual minor annoyances (and the associated years of life).
Of course, it depends somewhat on the framing of the question. For example, if you tortured me for half an hour and said “OK, I can either keep doing that for the next 50 years, or I can stop doing that and annually annoy you mildly for the rest of your immortal life,” I would definitely choose the latter. (Really, I’d probably agree to anything that included stopping the torture and didn’t violate some sacred value of mine, and quite likely I’d agree to most things that did violate my sacred values. Pain is like that.)
Yosarian2 keeps framing the question in terms of “what would you choose?” rather than “what would leave you better off?”, and then responding to selections of torture (which make sense in EY’s framing) with incredulity that anyone would actually choose torture.
At some point, fighting over the framing of the problem isn’t worth my time: f they insist on asking a (relatively trivial) epistemic question about my choices, and insist on ignoring the (more interesting) question of what would leave me better off, at some point I just decide to answer the question they asked and be done with it.
This is similar to my response to many trolley questions: faced with that choice, what I would actually do is probably hesitate ineffectually, allowing the 5 people to die. But the more interesting question is what I believe I ought to do.
Well… ideally, what you choose is what would leave you better off, and is chosen with this in mind. What you do ought to be what you ought to do, and what you ought to do ought to be what you do. Anything out of line with this either damages you unnecessarily or acknowledges that the you that might have desired the ought-choice is dead.
Yes, ideally, I would choose what would leave me better off, and what I do ought to be what I ought to do, and what I ought to do ought to be what I do. Also, I ought to do what I ought to do, and various other formulations of this thought. And yes, not doing what I ought to do has negative consequences relative to doing what I ought to do, which is precisely what makes what I ought to do what I ought to do in the first place.
the you that might have desired the ought-choice is dead.
This, on the other hand, makes no sense to me at all.
I’m also the sort of person who believes he has been dead for billions of years. Basically—if someone exists at some point and does not at another, they have died. We change over time; we throw off a chain of subtly different dead selves.
What that says to me is that most of us would not trade any amount of “occasional minor annoyances” for any amount of “horrific torture”. The two things might both be “bad”, but that doesn’t mean that X annoyances=Y torture, any more then (for positive values) X pleasure=Y freedom. (The initial problem helped disguise that by implying that the torture was going to happen to someone else, but in utilitarian terms that shouldn’t matter.)
When you can point to situations where “shutting up and multiplying” doesn’t seem to fit our actual values, then perhaps the simplistic kind of utilitarian moral system that allows you to do that kind of math is simply not a good description of our actual value system, at least not in all cases.
So, consider the general case of an ordered pair (X,Y) such that given a choice between X right now, and Y once a year for the next (insert arbitrarily large number here) years, most people would probably choose Y.
Where (X=”50 years of horrific torture”, Y= “a minor annoyance”), your reasoning leads you to conclude that most of us would not accept any amount of X in exchange for any amount of Y.
Where (X=”spending fifty thousand dollars”, Y=”spending a dollar”), would you similarly conclude that most of us would not accept any amount of fifty thousand dollars in exchange for any amount of dollars? I hope not, because that’s clearly false.
I conclude that your reasoning is not quite right.
All of that said, I agree that a simple utilitarian moral system doesn’t properly describe our actual value system in all cases.
The difference is that “dollars” and “fifty thousand dollars” are obviously equivalent and interchangeable units. 50,000 dollars equals one “fifty thousand dollars”, obviously.
I don’t think that any amount of “occasional minor annoyances” are equivalent or interchangeable with “a long period of horrific torture”. They aren’t even in the same catorgy, IMHO.
So, consider the general case of an ordered pair (X,Y) such that given a choice between X right now, and Y once a year for the next (insert arbitrarily large number here) years, most people would probably choose Y.
I think the time order here (torture now or annoyances later) may be another factor that is distracting us from the point, so let’s drop that.
Let’s say that you know, for a fact, that you will live for the next billion years. Now say that you have to choose between either having a very minor annoyance once a year for the next billion years, or instead having 50 years of horrific torture happen to you at some random point in the future within the next billion years. Personally, I would still choose the annoyances, rather then put my future-self through that horrific torture.
The difference is that “dollars” and “fifty thousand dollars” are obviously equivalent and interchangeable units.
Yes, I agree that the equivalence is far more obvious in this example.
I don’t think that any amount of “occasional minor annoyances” are equivalent or interchangeable with “a long period of horrific torture”.
You are welcome to believe that. It doesn’t follow from the premise you seemed earlier to be concluding it from, though. If you’re simply asserting it, that’s fine.
And, yes, given the choice you describe, I would probably make the same choice.
We’re comparing two values specifically here; torture vs minor annoyances. In that case, then yes, I would rather spend X money preventing someone from being tortured then spend the same X money protecting people from getting dust in their eye. The values, IHMO, are on fundamentally different levels.
Now, if it was “torture vs. saving lives”, or something like that, then maybe you could do the math.
It’s basically the same argument as the classic “how many people have to enjoy watching a gladiatorial combat on TV in order to morally justify forcing two people to fight for the death?” I don’t think you can do that math, because the values (enjoying watching a tv show vs. human life) are on fundamentally different levels.
If “shut up and multiply’ was always the right answer, if you could break everything down to utilitons and go with whatever provides the most utility for the most people, then the logical thing to do would be to create a trillion intelligent beings in the solar system who do nothing but enjoy pleasure all the time
Yes, that’s the right answer if you find the notion of one intelligent being who does nothing but wallow in pleasure as a net gain and you agree that there aren’t diminishing returns on creating such beings, and this doesn’t divert resources from any more important values.
But if you find the notion of even one such being repulsive or neutral, then you’re “shutting up and multiplying” over a non-positive number. In decision theory, utility quantifies how much you personally prefer certain scenarios, not pleasure.
I think we’re saying the same thing. Pleasure undoubtedly has some positive utility (or else we wouldn’t seek it out). But, like you said, you are diverting resources from “more important values”. That was, in a sense, the whole point of Eliezer’s story about the “Superhappys”.
So, by the same token, if we think “nobody should be tortured” is a more important value then “avoiding small amounts of annoyance”, then we should not sacrifice one for the other, not even for very large values of “avoiding annoyances”.
The only difference is that it’s more obvious when we’re talking about positive values (like pleasure) then when you’re talking about negative values (like avoiding someone being tortured).
I actually think that there is a fundamental problem with initial question that makes the answer different from the one Eliezer thinks.
If “shut up and multiply’ was always the right answer, if you could break everything down to utilitons and go with whatever provides the most utility for the most people, then the logical thing to do would be to create a trillion intelligent beings in the solar system who do nothing but enjoy pleasure all the time. But he’s made it clear that he doesn’t believe that, that he believes that humans have a wide variety of values that are irreducible, that are fundamentally different, and that if you give up some of those values to satisfy others, you lose something important, and we all end up worse off. When it comes to fundamentally different values, like “freedom”, that you can’t just say “X freedom is worth Y pleasure, so if you can create Y+1 pleasure you should give up on the concept of freedom”, not for any value of X and Y.
In this case, I would argue that the value “nobody should suffer from torture if it can be avoided” is a fundamentally different value from “it’s nice to avoid minor annoyances when possible”, and that you can’t simply break them down into utilitons and “do the math” like he suggests.
You’re conflating “utility” and “pleasure”. The aggregation function to use for multiple people is a question of some concern, but I think most of us would agree that maximizing utility is more complex than just wireheading.
Yes, that was my exact point.
Is there a positive utility value to pleasure? Obviously there is. But can you just “shut up and multiply” that, figure out that you get more utility from wireheading then from anything else, that and then decide that that’s the right choice? Obviously not, because human value is complex, and you can’t just give up one value to create utility somewhere else, not even for arbitrarily large amounts of pleasure.
I am arguing that the same is true for things with negative utility. Is there a negative utility value to a minor annoyance? Sure. But that doesn’t mean that we should give up other values, like “torturing people is wrong” in order to avoid annoyances, not even for arbitrarily large values of annoyances.
It seems to me we make value tradeoffs all the time, based on how much we value different things at that time.
I value living. I also value happiness. Given a choice between a shorter, happier life and a longer, less-happy life, I would choose one or the other. Either way, I’m trading one value off for another. You seem to be saying that I can’t do that. I don’t see how that claim even makes sense. If I can’t choose one or the other, what can I do?
Now, if you want to say that you happen to value avoiding torture in such a way that you won’t trade any amount of minor-annoyance-avoiding for any amount of torture… that your preferences are lexicographic in this regard… well, then yes, your values are significantly different from what Eliezer is talking about, and you (specifically you) can’t “do the math”.
But it sounds like you want to say that everyone is like that. What makes you think that?
I think that most people’s values are set up like that, which is why most people have such a strong negative instinctive reaction to the whole “dust motes vs torture” question. Perhaps not, though; I obviously can’t speak for everyone.
If you had to choose between being tortured horribly for the next 50 years, or having a minor annoyance happen to you once a year for the next (insert arbitrarily large number here) years, which would you pick?
Given that choice, I’d pick having a minor annoyance happen to me once a year for the next (insert arbitrarily large number here) years.
Are you so sure about that? If the number is large enough, it’s easily conceivable that at some point in your effectively infinite lifespan, you will come across a situation where that minor annoyance changes what might have been a perfectly good 50 years into 50 years of hell. So either you’ve wound up with mild scale insensitivity, a significant discount rate, or a rejection of additivity of negends.
I agree with you entirely.
I’m not saying I’d be better off having picked it. For the vast majority of numbers, I absolutely would not be. [EDIT: Well, assuming no knock-on effects from the torture, which EY’s initial formulation assumed.]
I’m saying it’s probably what I would, in fact, pick, if I were somehow in the epistemic state of being offered that choice. Yes, scale insensitivity and discounting play a role here, as does my confidence that I’m actually being offered an arbitrarily large number of annual minor annoyances (and the associated years of life).
Of course, it depends somewhat on the framing of the question. For example, if you tortured me for half an hour and said “OK, I can either keep doing that for the next 50 years, or I can stop doing that and annually annoy you mildly for the rest of your immortal life,” I would definitely choose the latter. (Really, I’d probably agree to anything that included stopping the torture and didn’t violate some sacred value of mine, and quite likely I’d agree to most things that did violate my sacred values. Pain is like that.)
Yosarian2 keeps framing the question in terms of “what would you choose?” rather than “what would leave you better off?”, and then responding to selections of torture (which make sense in EY’s framing) with incredulity that anyone would actually choose torture.
At some point, fighting over the framing of the problem isn’t worth my time: f they insist on asking a (relatively trivial) epistemic question about my choices, and insist on ignoring the (more interesting) question of what would leave me better off, at some point I just decide to answer the question they asked and be done with it.
This is similar to my response to many trolley questions: faced with that choice, what I would actually do is probably hesitate ineffectually, allowing the 5 people to die. But the more interesting question is what I believe I ought to do.
Well… ideally, what you choose is what would leave you better off, and is chosen with this in mind. What you do ought to be what you ought to do, and what you ought to do ought to be what you do. Anything out of line with this either damages you unnecessarily or acknowledges that the you that might have desired the ought-choice is dead.
Yes, ideally, I would choose what would leave me better off, and what I do ought to be what I ought to do, and what I ought to do ought to be what I do. Also, I ought to do what I ought to do, and various other formulations of this thought. And yes, not doing what I ought to do has negative consequences relative to doing what I ought to do, which is precisely what makes what I ought to do what I ought to do in the first place.
This, on the other hand, makes no sense to me at all.
Basically: if you’re not doing what you think you should be doing, you’re either screwing yourself or you’re not who you think you are.
Ah, I see. I think. I wouldn’t call that being dead, personally, but I can see why you do. I think.
I’m also the sort of person who believes he has been dead for billions of years. Basically—if someone exists at some point and does not at another, they have died. We change over time; we throw off a chain of subtly different dead selves.
Right, that’s what I figured you were using “dead” to mean.
(nods) Right. I think most people would.
What that says to me is that most of us would not trade any amount of “occasional minor annoyances” for any amount of “horrific torture”. The two things might both be “bad”, but that doesn’t mean that X annoyances=Y torture, any more then (for positive values) X pleasure=Y freedom. (The initial problem helped disguise that by implying that the torture was going to happen to someone else, but in utilitarian terms that shouldn’t matter.)
When you can point to situations where “shutting up and multiplying” doesn’t seem to fit our actual values, then perhaps the simplistic kind of utilitarian moral system that allows you to do that kind of math is simply not a good description of our actual value system, at least not in all cases.
So, consider the general case of an ordered pair (X,Y) such that given a choice between X right now, and Y once a year for the next (insert arbitrarily large number here) years, most people would probably choose Y.
Where (X=”50 years of horrific torture”, Y= “a minor annoyance”), your reasoning leads you to conclude that most of us would not accept any amount of X in exchange for any amount of Y.
Where (X=”spending fifty thousand dollars”, Y=”spending a dollar”), would you similarly conclude that most of us would not accept any amount of fifty thousand dollars in exchange for any amount of dollars? I hope not, because that’s clearly false.
I conclude that your reasoning is not quite right.
All of that said, I agree that a simple utilitarian moral system doesn’t properly describe our actual value system in all cases.
The difference is that “dollars” and “fifty thousand dollars” are obviously equivalent and interchangeable units. 50,000 dollars equals one “fifty thousand dollars”, obviously.
I don’t think that any amount of “occasional minor annoyances” are equivalent or interchangeable with “a long period of horrific torture”. They aren’t even in the same catorgy, IMHO.
I think the time order here (torture now or annoyances later) may be another factor that is distracting us from the point, so let’s drop that.
Let’s say that you know, for a fact, that you will live for the next billion years. Now say that you have to choose between either having a very minor annoyance once a year for the next billion years, or instead having 50 years of horrific torture happen to you at some random point in the future within the next billion years. Personally, I would still choose the annoyances, rather then put my future-self through that horrific torture.
Yes, I agree that the equivalence is far more obvious in this example.
You are welcome to believe that. It doesn’t follow from the premise you seemed earlier to be concluding it from, though. If you’re simply asserting it, that’s fine.
And, yes, given the choice you describe, I would probably make the same choice.
Really? Would you pay $10 billion to save one person from being tortured, assuming a remotely normal range of other things to spend the money on?
We’re comparing two values specifically here; torture vs minor annoyances. In that case, then yes, I would rather spend X money preventing someone from being tortured then spend the same X money protecting people from getting dust in their eye. The values, IHMO, are on fundamentally different levels.
Now, if it was “torture vs. saving lives”, or something like that, then maybe you could do the math.
It’s basically the same argument as the classic “how many people have to enjoy watching a gladiatorial combat on TV in order to morally justify forcing two people to fight for the death?” I don’t think you can do that math, because the values (enjoying watching a tv show vs. human life) are on fundamentally different levels.
Yes, that’s the right answer if you find the notion of one intelligent being who does nothing but wallow in pleasure as a net gain and you agree that there aren’t diminishing returns on creating such beings, and this doesn’t divert resources from any more important values.
But if you find the notion of even one such being repulsive or neutral, then you’re “shutting up and multiplying” over a non-positive number. In decision theory, utility quantifies how much you personally prefer certain scenarios, not pleasure.
I think we’re saying the same thing. Pleasure undoubtedly has some positive utility (or else we wouldn’t seek it out). But, like you said, you are diverting resources from “more important values”. That was, in a sense, the whole point of Eliezer’s story about the “Superhappys”.
So, by the same token, if we think “nobody should be tortured” is a more important value then “avoiding small amounts of annoyance”, then we should not sacrifice one for the other, not even for very large values of “avoiding annoyances”.
The only difference is that it’s more obvious when we’re talking about positive values (like pleasure) then when you’re talking about negative values (like avoiding someone being tortured).
The thing is, you should be able to trade off between your values, and that requires a currency of some sort, even if it’s only implicit.