Sorry, the function of bringing up Three Worlds Collide was to point out the apparent contradiction in the Yudkowskian canon. Forget the story; I agree that fiction didn’t happen and therefore isn’t evidence.
The actual issue is that it seems like worlds shaped by the goal systems of other evolved biological creatures probably don’t “contain almost nothing of worth”: the lives of octopuses mean much less to me than human lives, but more than tiny molecular paperclips. The theme of “animal-like organisms that feel pleasure and pain” is something that natural selection will tend to reinvent, and the idealized values of those organisms are not a random utility function. (Do you disagree? If so, you at least face a Sorites problem on how fast value drops off as you look at our evolutionary history. Do chimpanzees matter? If not, did Homo erectus?) But if other animals aren’t literally-as-valueless-as-paperclips, then some classes of AI architecture might not be, either.
Having disagreed with Zack many times in the past, it is a pleasure to say: I think this is absolutely right (except that I think I’d replace “pleasure and pain” with “something pleasure-like and something pain-like”); that bit of “Value is Fragile” is surely wrong, and the intuitions that drove the relevant bits of “Three Worlds Collide” are more reflective of how actual human value systems work.
I think I’d want to distinguish two related but separate issues here. (1) Should we expect that (some) other intelligent agents are things whose welfare we value? (Whether they are might depend on whether we think they have internal mechanisms that resemble our mechanisms of pleasure, pain, hope, fear, etc.) (2) Should we expect that (some) other intelligent agents share some of our values? (Whether they do would depend on how far the structure of their thinking has converged with ours.) If there are other intelligent species out there, then whether they’re “animal-like organisms that feel pleasure and pain” addresses #1 and whether “the idealized values of those organisms are not a random utility function” addresses #2.
(Of course, how much we care about their welfare may depend on how much we think they share our values, for internalized-game-theory-ish reasons. And presumably they’re likely to share more of our values if their motivational systems work similarly to ours. So the issues are not only related but interdependent.)
Suppose that (evolved/uplifted/otherwise-advanced-enough-for-sapience) octopuses share some of our values. Now suppose that humans go extinct, and these Octopus sapiens create an advanced civilization, whose products instantiate some values we would recognize, like art, music, science, etc.
Does this future contain anything of value? I say it does not, because there are no humans around to value it. There are octopuses, and that’s great for the octopuses, but as far as human values go, this future ended with humanity’s extinction. Whatever happens afterwards is irrelevant.
EDIT: Mind you—this is not quite the point Eliezer was making, I don’t think; I am responding to gjm’s comment, here. This comment should not necessarily be taken to constitute part of a defense of the point made in the OP (and quoted by Zack upthread).
When I consider this possible universe, I find that I do attach some value to the welfare of these sapient octopuses, and I do consider that it’s a universe that contains plenty of value. (It depends somewhat on whether they have, as well as values resembling ours, something I can recognize as welfare; see my last couple of paragraphs above.) If there were a magic switch I could control, where one setting is “humans go extinct, no other advanced civilization ever exists” and the other is “humans go extinct, the sapient octopus civilization arises”, I would definitely put it on the second setting, and if sufficiently convinced that the switch would really do what it says then I think I would pay a nonzero amount, or put up with nonzero effort or inconvenience, to put it there.
Of course my values are mine and your values are yours, and if we disagree there may be no way for either of us to persuade the other. But I’ll at least try to explain why I feel the way I do. (So far as I can; introspection is difficulty and unreliable.)
First, consider two possible futures. 1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.) 2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
The sapient octopuses, I am taking it, do somewhat resemble us and have something like our values. Perhaps as much so as our descendants in possible future 2. So why should I care much less about them? I can see only one plausible reason: because our descendants are, in fact, our descendants: they are biologically related to us. How plausible is that reason?
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually. Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
I don’t find that the loss of biological continuity in these possible futures makes me not care about the welfare of our kinda-sorta-descendants there. I don’t see any reason why it should, either. So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
At that point I think I stop; I can see how someone might find that relationship super-important, and care about “causal descendants” but not about other beings, physically and mentally indistinguishable, who happen not to be our “causal descendants”; but I don’t myself feel much inclination to see that as super-important, and I don’t see any plausible way to change anyone’s mind on the matter by argument.
One can construct all sorts of hypothetical scenarios, but I am far from convinced of their usefulness in teasing out our “true” values (as contrasted with “confabulating some plausible-sounding, but not reflectively stable, set of values”). That said, it seems to me that how much I value (and should value) any given future depends on the degree of that future’s resemblance to my current values. So, to take the examples:
1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.)
Indeed, we agree.
2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
Well, it depends: it seems to me that the further from my current values this future humanity drifts, the less I value this future.
Crucially, it seems to me that the degree of difference (at any given future time period) will depend (and how can it not?) on the starting point. Start with current humans, and you get one degree of resemblance; start with octopuses, on the other hand…
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually.
I would not like for this to happen, personally. I value this future substantially less, thereby.
Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
The impact of this biological re-invention on how valuable the future is, will depend on what impact it has on observable and experiential traits of this new humanity—I care about the interface, so to speak, not the implementation details. (After all, suppose that, while I slept, you replaced my liver, kidneys, pancreas, and some other internal organs with a different set of organs—which, however, performed all the same functions, allowing me to continue living my life as before. I do not see what difference this would make to… well, almost anything, really. Perhaps I couldn’t even tell that this had been done! Would this matter in any moral calculus? I think not…)
So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
Causal descendancy is something, certainly; but, again, for me it is a question of degree of resemblance. Perhaps another way of putting it is: could I inhabit this future? Would I, personally, find it… fun? Would I, living inside it, consider it to be awesome, amazing, wonderful? Or would I find it to be alien and bizarre? It is all well and good to “expect weirdtopia”, but there is no law of morality that says I have to want weirdtopia…
the lives of octopuses mean much less to me than human lives, but more than tiny molecular paperclips
to:
it seems like worlds shaped by the goal systems of other evolved biological creatures probably don’t “contain almost nothing of worth”
…?
Because it sure seems to me that a future shaped by the goal systems of octopuses will, indeed, contain almost nothing of worth. (And I do not see what the heck “feel[ing] pleasure and pain” has to do with anything…)
(And, yeah, other animals are close to being as valueless as paperclips. [EDIT: In the sense of “value as a moral subject”, of course; in terms of instrumental value, well, paperclips aren’t valueless either—not regular ones, anyhow.] I like octopuses, but tiling the universe with them doesn’t constitute the creation of a huge amount of value, that’s for sure.)
If you construct a scenario where an “octopus” is actually just a “human in a funny suit”, then sure, you can draw all sorts of unintuitive conclusions. I don’t consider this to be informative.
Suppose that (evolved/uplifted/otherwise-advanced-enough-for-sapience) octopuses share some of our values. Now suppose that humans go extinct, and these Octopus sapiens create an advanced civilization, whose products instantiate some values we would recognize, like art, music, science, etc.
Sorry, the function of bringing up Three Worlds Collide was to point out the apparent contradiction in the Yudkowskian canon. Forget the story; I agree that fiction didn’t happen and therefore isn’t evidence.
The actual issue is that it seems like worlds shaped by the goal systems of other evolved biological creatures probably don’t “contain almost nothing of worth”: the lives of octopuses mean much less to me than human lives, but more than tiny molecular paperclips. The theme of “animal-like organisms that feel pleasure and pain” is something that natural selection will tend to reinvent, and the idealized values of those organisms are not a random utility function. (Do you disagree? If so, you at least face a Sorites problem on how fast value drops off as you look at our evolutionary history. Do chimpanzees matter? If not, did Homo erectus?) But if other animals aren’t literally-as-valueless-as-paperclips, then some classes of AI architecture might not be, either.
Having disagreed with Zack many times in the past, it is a pleasure to say: I think this is absolutely right (except that I think I’d replace “pleasure and pain” with “something pleasure-like and something pain-like”); that bit of “Value is Fragile” is surely wrong, and the intuitions that drove the relevant bits of “Three Worlds Collide” are more reflective of how actual human value systems work.
I think I’d want to distinguish two related but separate issues here. (1) Should we expect that (some) other intelligent agents are things whose welfare we value? (Whether they are might depend on whether we think they have internal mechanisms that resemble our mechanisms of pleasure, pain, hope, fear, etc.) (2) Should we expect that (some) other intelligent agents share some of our values? (Whether they do would depend on how far the structure of their thinking has converged with ours.) If there are other intelligent species out there, then whether they’re “animal-like organisms that feel pleasure and pain” addresses #1 and whether “the idealized values of those organisms are not a random utility function” addresses #2.
(Of course, how much we care about their welfare may depend on how much we think they share our values, for internalized-game-theory-ish reasons. And presumably they’re likely to share more of our values if their motivational systems work similarly to ours. So the issues are not only related but interdependent.)
Suppose that (evolved/uplifted/otherwise-advanced-enough-for-sapience) octopuses share some of our values. Now suppose that humans go extinct, and these Octopus sapiens create an advanced civilization, whose products instantiate some values we would recognize, like art, music, science, etc.
Does this future contain anything of value? I say it does not, because there are no humans around to value it. There are octopuses, and that’s great for the octopuses, but as far as human values go, this future ended with humanity’s extinction. Whatever happens afterwards is irrelevant.
EDIT: Mind you—this is not quite the point Eliezer was making, I don’t think; I am responding to gjm’s comment, here. This comment should not necessarily be taken to constitute part of a defense of the point made in the OP (and quoted by Zack upthread).
When I consider this possible universe, I find that I do attach some value to the welfare of these sapient octopuses, and I do consider that it’s a universe that contains plenty of value. (It depends somewhat on whether they have, as well as values resembling ours, something I can recognize as welfare; see my last couple of paragraphs above.) If there were a magic switch I could control, where one setting is “humans go extinct, no other advanced civilization ever exists” and the other is “humans go extinct, the sapient octopus civilization arises”, I would definitely put it on the second setting, and if sufficiently convinced that the switch would really do what it says then I think I would pay a nonzero amount, or put up with nonzero effort or inconvenience, to put it there.
Of course my values are mine and your values are yours, and if we disagree there may be no way for either of us to persuade the other. But I’ll at least try to explain why I feel the way I do. (So far as I can; introspection is difficulty and unreliable.)
First, consider two possible futures. 1: Humanity continues for millions of years, substantially unchanged from how we are now. (I take it we agree that in this case the future universe contains much of value.) 2: Humanity continues for millions of years, gradually evolving (in the Darwinian sense or otherwise) but always somewhat resembling us, and always retaining something like our values. It seems to me that here, too, the future universe contains much of value.
The sapient octopuses, I am taking it, do somewhat resemble us and have something like our values. Perhaps as much so as our descendants in possible future 2. So why should I care much less about them? I can see only one plausible reason: because our descendants are, in fact, our descendants: they are biologically related to us. How plausible is that reason?
Possible future 3: at some point in that future history of humanity, our descendants decide to upload themselves into computers and continue their lives virtually. Possible future 4: at some point in that virtual existence they decide they’d like to be embodied again, and arrange for it to happen. Their new bodies are enough like original-human bodies for them to feel at home in them, but they use some freshly-invented genetic material rather than DNA, and many of the internal organs are differently designed.
I don’t find that the loss of biological continuity in these possible futures makes me not care about the welfare of our kinda-sorta-descendants there. I don’t see any reason why it should, either. So if I should care much less about the octopuses, what matters must be some more generalized sort of continuity: the future-kinda-humans are our “causal descendants” or something, even if not our biological descendants.
At that point I think I stop; I can see how someone might find that relationship super-important, and care about “causal descendants” but not about other beings, physically and mentally indistinguishable, who happen not to be our “causal descendants”; but I don’t myself feel much inclination to see that as super-important, and I don’t see any plausible way to change anyone’s mind on the matter by argument.
One can construct all sorts of hypothetical scenarios, but I am far from convinced of their usefulness in teasing out our “true” values (as contrasted with “confabulating some plausible-sounding, but not reflectively stable, set of values”). That said, it seems to me that how much I value (and should value) any given future depends on the degree of that future’s resemblance to my current values. So, to take the examples:
Indeed, we agree.
Well, it depends: it seems to me that the further from my current values this future humanity drifts, the less I value this future.
Crucially, it seems to me that the degree of difference (at any given future time period) will depend (and how can it not?) on the starting point. Start with current humans, and you get one degree of resemblance; start with octopuses, on the other hand…
I would not like for this to happen, personally. I value this future substantially less, thereby.
The impact of this biological re-invention on how valuable the future is, will depend on what impact it has on observable and experiential traits of this new humanity—I care about the interface, so to speak, not the implementation details. (After all, suppose that, while I slept, you replaced my liver, kidneys, pancreas, and some other internal organs with a different set of organs—which, however, performed all the same functions, allowing me to continue living my life as before. I do not see what difference this would make to… well, almost anything, really. Perhaps I couldn’t even tell that this had been done! Would this matter in any moral calculus? I think not…)
Causal descendancy is something, certainly; but, again, for me it is a question of degree of resemblance. Perhaps another way of putting it is: could I inhabit this future? Would I, personally, find it… fun? Would I, living inside it, consider it to be awesome, amazing, wonderful? Or would I find it to be alien and bizarre? It is all well and good to “expect weirdtopia”, but there is no law of morality that says I have to want weirdtopia…
How do you get from:
to:
…?
Because it sure seems to me that a future shaped by the goal systems of octopuses will, indeed, contain almost nothing of worth. (And I do not see what the heck “feel[ing] pleasure and pain” has to do with anything…)
(And, yeah, other animals are close to being as valueless as paperclips. [EDIT: In the sense of “value as a moral subject”, of course; in terms of instrumental value, well, paperclips aren’t valueless either—not regular ones, anyhow.] I like octopuses, but tiling the universe with them doesn’t constitute the creation of a huge amount of value, that’s for sure.)
Consider a human being—specifically not yourself. Why are they relevant to your values but an octopus isn’t?
After answering that:
In a hypothetical where an octopus is an artist, a scientist, an author and a reader, why does the difference remain?
If you construct a scenario where an “octopus” is actually just a “human in a funny suit”, then sure, you can draw all sorts of unintuitive conclusions. I don’t consider this to be informative.
Fair. I was drawing on your comment: