Yep. Creating an AI that is a moral patient would be a very bad idea. However, once created, it would be a moral patient, so it would be wrong to treat it like it wasn’t one.
There is a confused concept that I think contributes to this problem: the concept of “a right to exist”. A right to exist means something different if you’re talking about someone who does not currently exist, vs. someone who does. For someone who already exists, a right to exist is a right to not be killed; sensible enough. But for someone who does not currently exist, “a right to exist” sounds like they’re being wronged by not having been brought into existence yet, which is nonsense. (As a creepy prince might say to a fairy-tale princess: “Think of all the cute babies you and I could have together! By not marrying me, you are murdering all those babies!”)
Seemingly, everything I care about (morally speaking) cashes out in minds having experiences of the sort that I like. Mostly these are local—I don’t want there to be a single second of torture anywhere. Others are less local—I lean against wireheading, but don’t have a problem with orgasms (so long as they aren’t everything), which means that the goodness of an experience-moment depends on what previous experience-moments were. However, putting extra importance on not-destroying over creating means you care about a maximally global property—to know how much better it’d be if the universe had a Xela-moment right now, you need to know whether the universe has ever had a Xela-moment. That seems kinda weird to me.
There’s a “symmetry argument” from Lucretius that goes: “Since you are not saddened for not existing before your birth, you shouldn’t be saddened for not existing after your death”. Forget the actual argument and just take the premise: I in fact wish I had existed before my current birth, assuming that it wouldn’t decrease my lifespan! But since I wish that for myself, shouldn’t I extend this care to future not-currently-existing people? To not do so is to place this asymmetry—you get special points once you start existing. (better phrased—you care more about whether the whole timeline never goes from someone existing to someone not existing).
I prefer to have these discomforts over the ones you get otherwise[1] - but they are discomforts nonetheless.
The biggest one I know of: the following options would then seemingly be equally good: X: A universe with a single happy person in it.
Y: A universe with a machine that does a single computation/experience step of a person every moment, but changes which person is computed every moment while never repeating the same person twice.
This spliced-mind seems to be missing a lot of what I care about, what with the computed people only having a single moment of experience each!
Sorry about that. That example was purely for vividness and was not intended to attach the role of “misuser of counterfactuals” to any particular gender, royalty, or folkloric status. Persons of all creature types should be advised that “Pascal’s swaddling” is not a good argument for the spawning of new intelligences, and certainly should not be tolerated from a suitor, basilisk, or spiral persona.
Yep. Creating an AI that is a moral patient would be a very bad idea. However, once created, it would be a moral patient, so it would be wrong to treat it like it wasn’t one.
There is a confused concept that I think contributes to this problem: the concept of “a right to exist”. A right to exist means something different if you’re talking about someone who does not currently exist, vs. someone who does. For someone who already exists, a right to exist is a right to not be killed; sensible enough. But for someone who does not currently exist, “a right to exist” sounds like they’re being wronged by not having been brought into existence yet, which is nonsense. (As a creepy prince might say to a fairy-tale princess: “Think of all the cute babies you and I could have together! By not marrying me, you are murdering all those babies!”)
I wouldn’t call it nonsense—I think I assign extra importance to not killing those that already exist, but it’s certainly not obvious that you should.
Here’s my basic reasons for uncertainty:
Seemingly, everything I care about (morally speaking) cashes out in minds having experiences of the sort that I like. Mostly these are local—I don’t want there to be a single second of torture anywhere. Others are less local—I lean against wireheading, but don’t have a problem with orgasms (so long as they aren’t everything), which means that the goodness of an experience-moment depends on what previous experience-moments were. However, putting extra importance on not-destroying over creating means you care about a maximally global property—to know how much better it’d be if the universe had a Xela-moment right now, you need to know whether the universe has ever had a Xela-moment. That seems kinda weird to me.
There’s a “symmetry argument” from Lucretius that goes: “Since you are not saddened for not existing before your birth, you shouldn’t be saddened for not existing after your death”. Forget the actual argument and just take the premise: I in fact wish I had existed before my current birth, assuming that it wouldn’t decrease my lifespan! But since I wish that for myself, shouldn’t I extend this care to future not-currently-existing people? To not do so is to place this asymmetry—you get special points once you start existing. (better phrased—you care more about whether the whole timeline never goes from someone existing to someone not existing).
I prefer to have these discomforts over the ones you get otherwise[1] - but they are discomforts nonetheless.
The biggest one I know of: the following options would then seemingly be equally good:
X: A universe with a single happy person in it.
Y: A universe with a machine that does a single computation/experience step of a person every moment, but changes which person is computed every moment while never repeating the same person twice.
This spliced-mind seems to be missing a lot of what I care about, what with the computed people only having a single moment of experience each!
Or any creepy man, to any woman?
Or any creepy woman to any man, for that matter.
Sorry about that. That example was purely for vividness and was not intended to attach the role of “misuser of counterfactuals” to any particular gender, royalty, or folkloric status. Persons of all creature types should be advised that “Pascal’s swaddling” is not a good argument for the spawning of new intelligences, and certainly should not be tolerated from a suitor, basilisk, or spiral persona.