The issue here is that the setup has a firm answer—A - but if it were tweaked ever so slightly, the whole preferences would change.
First of all, consider the following:
A’: there is one genuine simulation, and the other 99 are simple copies of that one. Soon, all the copies will be stopped, but the the true simulation will continue.
There are essentially no practical ways of distinguishing A from A’. So we should reason as if A’ were correct, in which case nothing is lost from the turning off.
However, if we allow any divergence between the copies, then this is a whole other kettle of barreled fish. Now there are at least a hundred copies of each person, and they are distinct: they think slightly differently, behave slightly differenctly, have slightly different goals, etc… And this divergence will only grow with time.
There are three ways I can think of addressing this:
1) Once copies diverge at all, they are different people. So any divergence results in multiplying by a hundred the amount of people in the universe. Hence, if we suspect divergence, we should treat each copy as totally distinct.
2) An information theoretic approach: given one individual, how much new information is needed to completely descibe another copy. Under this, a slightly divergent copy counts as much less than a completely new individual, while a very divergent copy is nearly entriely different.
Both these have drawbacks − 1) gives too much weight to insignificant and minute changes, and 2) implies that people in general are of nothing like approximately equal worth: only extreme exceentrics count as complete people, others are of much less importance. So I suggest a compromise:
3) An information theoretical cut off: two slightly divergent copies count as only slightly more than a single person, but once this divergence rises above some critical amount, the two are treated as entirely seperate individuals.
What is the relevance of this to the original problem? Well, as I said, the original problem has a clear-cut solution, but it is very close to the different situation I described. So, bearing in mind imperfect information, issues of trust and uncertainty, and trying to find similar situations to similar problems, I think we should treat the set-up as the “slightly divergent one”. In this situation, A still dominates, but the relative attraction of B rises as the divergence grows.
EDIT: didn’t explain the cutoff properly; I was meaning a growing measure of difference that hits a maximum at the cutoff point, and doesn’t grow any further. My response here gives an example of this.
Bite the bullet and select 2. There doesn’t really seem to be anything inherently wrong with that, while 3 seems ad hoc and thus bad.
You seem to underestimate the difference between two human minds, let alone other minds. Additional benefit from method 2 is that it explains why human suffering is more “wrong” than to some ants suffering. This I guess is intuitive way most here think.
3 seems approximately how we deal with people in reality—we say things like “A is so like B” or “A is so unlike B” without thinking that A and B are any less seperate individuals with distinct rights, legal and moral statuses. It’s only when A and B get much too close in their reactions, that we flip into a different mode and wonder whether they are trully seperate.
Since this is the moral intuition, I see no compelling reason to discard it. It doesn’t seem to contradict any major results, I haven’t yet seen thought experiments where it becomes ridiculous, and it doesn’t over-burden our decision process.
If any of my statements there turn out to be wrong, then I would consider embrasing 2).
Actually, I updated thanks to reading this paper by Bostrom, so I gotta rephrase stuff.
First, two identical people living in separate simulations are just as much separate people as any other separate people are separate people. It doesn’t matter if there exist an identical replica somewhere else, the value of this person in particular doesn’t decrease tinyest bit. They’re distinct but identical.
Second, uniqueness of their identity decreases as there are more people like them, as you described with option #2. However, this property is not too interesting, since it has nothing to do with their personal experiences.
So what we get is this: Simulations together hold 100x our experiences, and we’re either offered a deal that kills 99x of us, or the one that 99% certainly kills us. Both have same expected utility. On both cases, we cease to be with 99% certainty. Both have the same negative expected utility.
But the interesting thing happens when we try to ensure continued flow of us existing, since by some magic, it seems to favor A to B, when both seem to be perfectly equal. I’m kinda feeling that problematic nature of handling divergence comes from irrational nature of this tendency to value continued flow of existence. But dunno.
First, two identical people living in separate simulations are just as much separate people as any other separate people are separate people. It doesn’t matter if there exist an identical replica somewhere else, the value of this person in particular doesn’t decrease tinyest bit. They’re distinct but identical.
The problem with this is that preference is for choosing actions, and actions can’t be about specific people only, they are about the whole reality. The question of how much you value a person only makes sense in the context of a specific way of combining valuations of individual people into valuations of reality.
I’ve read the paper, and disagree with it (one flippant way of phrasing my disagreement is to enquire as to whether reflections in mirrors have identical moral status). See the beggining of my first post for a better objection.
The problem with option 3 is that its fundamentally intuitionist, with arbitrary
cutoffs distinguishing “real” individuals from copies. I mean, is there really
such a big difference between cutoff - .001 difference and cutoff + .001
difference? There isn’t. Unless you can show that there’s a qualitative
difference that occurs when that threshold is crossed, its much more elegant to
look at a distinction between options 1 and 2 without trying to artificially
shift the boundary between the two.
Let D be some objective measure of distance (probably to do with Kologomorov complexity) between individuals. Let M be my moral measure of distance, and assume the cut-off is 1.
Then I would set M(a,b) = D(a,b) whenever D(a,b) < 1, and M(a,b) = 1 whenever D(a,b) >= 1. The discontinuity is in the derivative, not the value.
That doesn’t resolve quanticle’s objection. Your cutoff still suggests that a reasonably individualistic human is just as valuable as, say, the only intelligent alien being in the universe. Would you agree with that conclusion?
No. I grant special status to exceedingly unique minds, and to the last few of a given species.
But human minds are very similar to each other, and granting different moral status to different humans is a very dangerous game. Here, I am looking at the practical effects of moral systems (Eliezer’s post on “running on corrupted hardware” is relevant). The thoeretical gains of treating humans as having varrying moral status are small; the practical risks are huge (especially as our societies, though cash, reputation and other factors, is pretty good at distinguishing between people without having to further grant them different moral status).
One cannot argue: “I agree with moral system M, but M has consequence S, and I disagree with S”. Hence I cannot agree with granting people different moral status, once they are sufficiently divergent.
Stuart’s option 3 says that the difference between “cutoff - .001” and “cutoff + .001″ is .001 (as opposed to .002 that it wold be if you value the divergence directly). i.e. cutoff is the point at which your distance metric saturates. It’s a nonlinearity, but not a discontinuity.
The issue here is that the setup has a firm answer—A - but if it were tweaked ever so slightly, the whole preferences would change.
First of all, consider the following:
A’: there is one genuine simulation, and the other 99 are simple copies of that one. Soon, all the copies will be stopped, but the the true simulation will continue.
There are essentially no practical ways of distinguishing A from A’. So we should reason as if A’ were correct, in which case nothing is lost from the turning off.
However, if we allow any divergence between the copies, then this is a whole other kettle of barreled fish. Now there are at least a hundred copies of each person, and they are distinct: they think slightly differently, behave slightly differenctly, have slightly different goals, etc… And this divergence will only grow with time.
There are three ways I can think of addressing this:
1) Once copies diverge at all, they are different people. So any divergence results in multiplying by a hundred the amount of people in the universe. Hence, if we suspect divergence, we should treat each copy as totally distinct.
2) An information theoretic approach: given one individual, how much new information is needed to completely descibe another copy. Under this, a slightly divergent copy counts as much less than a completely new individual, while a very divergent copy is nearly entriely different.
Both these have drawbacks − 1) gives too much weight to insignificant and minute changes, and 2) implies that people in general are of nothing like approximately equal worth: only extreme exceentrics count as complete people, others are of much less importance. So I suggest a compromise:
3) An information theoretical cut off: two slightly divergent copies count as only slightly more than a single person, but once this divergence rises above some critical amount, the two are treated as entirely seperate individuals.
What is the relevance of this to the original problem? Well, as I said, the original problem has a clear-cut solution, but it is very close to the different situation I described. So, bearing in mind imperfect information, issues of trust and uncertainty, and trying to find similar situations to similar problems, I think we should treat the set-up as the “slightly divergent one”. In this situation, A still dominates, but the relative attraction of B rises as the divergence grows.
EDIT: didn’t explain the cutoff properly; I was meaning a growing measure of difference that hits a maximum at the cutoff point, and doesn’t grow any further. My response here gives an example of this.
Bite the bullet and select 2. There doesn’t really seem to be anything inherently wrong with that, while 3 seems ad hoc and thus bad.
You seem to underestimate the difference between two human minds, let alone other minds. Additional benefit from method 2 is that it explains why human suffering is more “wrong” than to some ants suffering. This I guess is intuitive way most here think.
Edit: Fixed a lot of typos
3 seems approximately how we deal with people in reality—we say things like “A is so like B” or “A is so unlike B” without thinking that A and B are any less seperate individuals with distinct rights, legal and moral statuses. It’s only when A and B get much too close in their reactions, that we flip into a different mode and wonder whether they are trully seperate.
Since this is the moral intuition, I see no compelling reason to discard it. It doesn’t seem to contradict any major results, I haven’t yet seen thought experiments where it becomes ridiculous, and it doesn’t over-burden our decision process.
If any of my statements there turn out to be wrong, then I would consider embrasing 2).
Actually, I updated thanks to reading this paper by Bostrom, so I gotta rephrase stuff.
First, two identical people living in separate simulations are just as much separate people as any other separate people are separate people. It doesn’t matter if there exist an identical replica somewhere else, the value of this person in particular doesn’t decrease tinyest bit. They’re distinct but identical.
Second, uniqueness of their identity decreases as there are more people like them, as you described with option #2. However, this property is not too interesting, since it has nothing to do with their personal experiences.
So what we get is this: Simulations together hold 100x our experiences, and we’re either offered a deal that kills 99x of us, or the one that 99% certainly kills us. Both have same expected utility. On both cases, we cease to be with 99% certainty. Both have the same negative expected utility.
But the interesting thing happens when we try to ensure continued flow of us existing, since by some magic, it seems to favor A to B, when both seem to be perfectly equal. I’m kinda feeling that problematic nature of handling divergence comes from irrational nature of this tendency to value continued flow of existence. But dunno.
The problem with this is that preference is for choosing actions, and actions can’t be about specific people only, they are about the whole reality. The question of how much you value a person only makes sense in the context of a specific way of combining valuations of individual people into valuations of reality.
I’ve read the paper, and disagree with it (one flippant way of phrasing my disagreement is to enquire as to whether reflections in mirrors have identical moral status). See the beggining of my first post for a better objection.
The problem with option 3 is that its fundamentally intuitionist, with arbitrary cutoffs distinguishing “real” individuals from copies. I mean, is there really such a big difference between cutoff - .001 difference and cutoff + .001 difference? There isn’t. Unless you can show that there’s a qualitative difference that occurs when that threshold is crossed, its much more elegant to look at a distinction between options 1 and 2 without trying to artificially shift the boundary between the two.
Didn’t phrase clearly what I meant by cut-off.
Let D be some objective measure of distance (probably to do with Kologomorov complexity) between individuals. Let M be my moral measure of distance, and assume the cut-off is 1.
Then I would set M(a,b) = D(a,b) whenever D(a,b) < 1, and M(a,b) = 1 whenever D(a,b) >= 1. The discontinuity is in the derivative, not the value.
That doesn’t resolve quanticle’s objection. Your cutoff still suggests that a reasonably individualistic human is just as valuable as, say, the only intelligent alien being in the universe. Would you agree with that conclusion?
No. I grant special status to exceedingly unique minds, and to the last few of a given species.
But human minds are very similar to each other, and granting different moral status to different humans is a very dangerous game. Here, I am looking at the practical effects of moral systems (Eliezer’s post on “running on corrupted hardware” is relevant). The thoeretical gains of treating humans as having varrying moral status are small; the practical risks are huge (especially as our societies, though cash, reputation and other factors, is pretty good at distinguishing between people without having to further grant them different moral status).
One cannot argue: “I agree with moral system M, but M has consequence S, and I disagree with S”. Hence I cannot agree with granting people different moral status, once they are sufficiently divergent.
Stuart’s option 3 says that the difference between “cutoff - .001” and “cutoff + .001″ is .001 (as opposed to .002 that it wold be if you value the divergence directly). i.e. cutoff is the point at which your distance metric saturates. It’s a nonlinearity, but not a discontinuity.