Utilitarianism implies that if we build an AI that successfully maximizes utility/value, we should be ok with it replacing us
Under the premise of successful alignment, the “replacement” only happens if it’s the right thing to do, which doesn’t seem at all likely (depending on what “replacing” means). Optimization of the world doesn’t destructively overwrite it with optimal stuff, it instead moves it in optimal ways starting with its actual initial state, which happens to include humanity.
There’s no reason for me to think that my personal preferences (e.g. that my descendants exist) are related to the “right thing to do”, and so there’s no reason for me to think that optimizing the world for the “right things” will fulfil my preference.
I think most people share similar preferences to me when it comes to their descendants existing, which is why I expect my sentiment to be relatable and hope to collaborate with others on preventing humanity’s end.
There’s no reason for me to think that my personal preferences (e.g. that my descendants exist) are related to the “right thing to do”, and so there’s no reason for me to think that optimizing the world for the “right things” will fulfil my preference.
This, and several of the passages in your original post such as, “I agree such a definition of moral value would be hard to justify,” seem to imply some assumption of moral realism that I sometimes encounter as well, but have never really found convincing arguments for. I would say that the successionists you’re talking to are making a category error, and I would not much trust their understanding of ‘should’-ness outside normal day-to-day contexts.
I can’t really imagine a scenario where I “should” or would be ok with currently-existing-humans going extinct, though that doesn’t mean none could exist. I can, however, imagine a future where humanity chooses to cease (most?) natural biological reproduction in favor of other methods of bringing new life into the world, whether biological or artificial, which I could endorse (especially if we become biologically or otherwise immortal as individuals). I can further imagine being ok with those remaining biological humans each changing (gradually or suddenly) various aspects of their bodies, their minds, and the substrates their minds run on, until they are no longer meat-based and/or no longer ‘human’ in various ways most people currently understand the term.
This, and several of the passages in your original post such as, “I agree such a definition of moral value would be hard to justify,” seem to imply some assumption of moral realism that I sometimes encounter as well, but have never really found convincing arguments for. I would say that the successionists you’re talking to are making a category error, and I would not much trust their understanding of ‘should’-ness outside normal day-to-day contexts.
I broadly agree.
I am indeed being a bit sloppy with the moral language in my post. What I mean to say is something like “insofar as you’re trying to describe a moral realist position with a utility function to be optimized for, it’d be hard to justify valuing your specific likeness”.
In a similar fashion, I prefer and value my family more than your family but it’d be weird for me to say that you also should prefer my family to your own family.
However, I expect our interests and preferences to align when it comes to preferring that we have the right to prefer our own families, or preferring that our species exists.
(Meta: I am extremely far from an expert on moral philosophy or philosophy in general, I do aspire to improve how rigorously I am able to articulate my positions.)
I expect it’s the right thing to do, for the world to let you pursue your personal preferences. Values in decision theory sense are about the way the world develops, not about the way any given person lives their life, any more than laws of physics are about the way a person lives their life, even though it’s certainly going to be in strict accordance with the laws of physics.
So decision theoretic values can easily stipulate that people should keep their differing preferences, have conflicts and disagreements, make mistakes and cause disasters, and so on. And that people with these properties should be part of the world simply because they already are, and not because they necessarily needed to exist in this form, with these preferences and with these conflicts and disagreements, if there was no humanity in the world from the outset.
The right thing to do is subjective, and what we’re trying for is to get our actual, individual or collective human values into AI systems. Specifying that is TBD and I think pretty confusing and hard. However, it seems that if we are all killed off and replaced, we would presumably have failed, since that is not something most of us want.
Nina is worried not just about humans getting killed and replaced, but also about humans not being allowed to have unenhanced children. It seems plausible that most humans, after reflection, would endorse some kind of “successionist” philosophy/ideology, and decide that intentionally creating an unenhanced human constitutes a form of child abuse (e.g., due to risk of psychological tendency to suffer, or having a much worse life on expectation than what’s possible). It seems reasonable for Nina to worry about this, if she thinks her own values (current or eventual or actual) are different.
(btw i expect we’ll really want enhanced humans to have the capacity to suffer, because we have preferences around future people being able to experience the kinds of feelings we experience when we read stories, including very sad stories. Some suffering is reflectively endorsed and we enjoy it/wouldn’t want it to not happen; and it seems fine to want new humans and enhanced current humans to also have it, although maybe with more access to some control over it.)
Certainly an aligned AI can be a serious threat if you have sufficiently unusual values relative to whoever does the aligning. That worries me a lot—I think many possible “positive” outcomes are still somewhat against my interests and are also undemocratic, stripping agency from many people. However, if this essay were capable of convincing “humanity” that they shouldn’t value enhancement, CEV should already have that baked in?
No, because power/influence dynamics could be very different in CEV compared to the current world and it seems reasonable to distrust CEV in principle or in practice, and/or CEV is sensitive to initial conditions implying a lot of leverage to influencing opinions before it starts.
Under the premise of successful alignment, the “replacement” only happens if it’s the right thing to do, which doesn’t seem at all likely (depending on what “replacing” means). Optimization of the world doesn’t destructively overwrite it with optimal stuff, it instead moves it in optimal ways starting with its actual initial state, which happens to include humanity.
There’s no reason for me to think that my personal preferences (e.g. that my descendants exist) are related to the “right thing to do”, and so there’s no reason for me to think that optimizing the world for the “right things” will fulfil my preference.
I think most people share similar preferences to me when it comes to their descendants existing, which is why I expect my sentiment to be relatable and hope to collaborate with others on preventing humanity’s end.
This, and several of the passages in your original post such as, “I agree such a definition of moral value would be hard to justify,” seem to imply some assumption of moral realism that I sometimes encounter as well, but have never really found convincing arguments for. I would say that the successionists you’re talking to are making a category error, and I would not much trust their understanding of ‘should’-ness outside normal day-to-day contexts.
In other words: it sounds like you don’t want to be replaced under any conditions you can foresee.. You have judged. What else is there?
I can’t really imagine a scenario where I “should” or would be ok with currently-existing-humans going extinct, though that doesn’t mean none could exist. I can, however, imagine a future where humanity chooses to cease (most?) natural biological reproduction in favor of other methods of bringing new life into the world, whether biological or artificial, which I could endorse (especially if we become biologically or otherwise immortal as individuals). I can further imagine being ok with those remaining biological humans each changing (gradually or suddenly) various aspects of their bodies, their minds, and the substrates their minds run on, until they are no longer meat-based and/or no longer ‘human’ in various ways most people currently understand the term.
I broadly agree.
I am indeed being a bit sloppy with the moral language in my post. What I mean to say is something like “insofar as you’re trying to describe a moral realist position with a utility function to be optimized for, it’d be hard to justify valuing your specific likeness”.
In a similar fashion, I prefer and value my family more than your family but it’d be weird for me to say that you also should prefer my family to your own family.
However, I expect our interests and preferences to align when it comes to preferring that we have the right to prefer our own families, or preferring that our species exists.
(Meta: I am extremely far from an expert on moral philosophy or philosophy in general, I do aspire to improve how rigorously I am able to articulate my positions.)
I expect it’s the right thing to do, for the world to let you pursue your personal preferences. Values in decision theory sense are about the way the world develops, not about the way any given person lives their life, any more than laws of physics are about the way a person lives their life, even though it’s certainly going to be in strict accordance with the laws of physics.
So decision theoretic values can easily stipulate that people should keep their differing preferences, have conflicts and disagreements, make mistakes and cause disasters, and so on. And that people with these properties should be part of the world simply because they already are, and not because they necessarily needed to exist in this form, with these preferences and with these conflicts and disagreements, if there was no humanity in the world from the outset.
The right thing to do is subjective, and what we’re trying for is to get our actual, individual or collective human values into AI systems. Specifying that is TBD and I think pretty confusing and hard. However, it seems that if we are all killed off and replaced, we would presumably have failed, since that is not something most of us want.
Nina is worried not just about humans getting killed and replaced, but also about humans not being allowed to have unenhanced children. It seems plausible that most humans, after reflection, would endorse some kind of “successionist” philosophy/ideology, and decide that intentionally creating an unenhanced human constitutes a form of child abuse (e.g., due to risk of psychological tendency to suffer, or having a much worse life on expectation than what’s possible). It seems reasonable for Nina to worry about this, if she thinks her own values (current or eventual or actual) are different.
(btw i expect we’ll really want enhanced humans to have the capacity to suffer, because we have preferences around future people being able to experience the kinds of feelings we experience when we read stories, including very sad stories. Some suffering is reflectively endorsed and we enjoy it/wouldn’t want it to not happen; and it seems fine to want new humans and enhanced current humans to also have it, although maybe with more access to some control over it.)
Certainly an aligned AI can be a serious threat if you have sufficiently unusual values relative to whoever does the aligning. That worries me a lot—I think many possible “positive” outcomes are still somewhat against my interests and are also undemocratic, stripping agency from many people.
However, if this essay were capable of convincing “humanity” that they shouldn’t value enhancement, CEV should already have that baked in?
No, because power/influence dynamics could be very different in CEV compared to the current world and it seems reasonable to distrust CEV in principle or in practice, and/or CEV is sensitive to initial conditions implying a lot of leverage to influencing opinions before it starts.