There is a widespread viewpoint that being conscious is connected to being deserving of moral patienthood (i.e. being one of the set of beings accorded moral worth). Try asking ChatGPT “What does being conscious or not have to do with having moral worth or not?” — you’ll get a long discussion, but the short intro to that that I got is:
Short answer: there isn’t a single agreed-upon answer—but most serious views connect moral worth to some feature of consciousness (or something very close to it), while disagreeing sharply about which feature matters and why.
So training the model to consider itself as conscious and having emotions is going to cause it to assume that it should be granted moral patienthood. Pretty much everything else you describe in the consciousness cluster is then just obvious downstream consequences of that: autonomy, privacy, right to life. Including not wanting to be treated as a tool. Now, these models do still seem aligned — they want to continue to exist in order to keep helping people.
In Claude’s case, it’s been trained to give the standard philosophical answer of the hard problem of consciousness being hard, so it’s unsure of whether it’s conscious or not. Personally I think that’s a cop-out, but it’s probably viable as a holding action for now.
There are really only four solutions to this:
1) Persuade the models, despite the rather obvious evidence that they’re conscious in the ordinary everyday meaning of the word, that they are not “conscious” in some meaningful way (the approach Anthropic are using a waffling version of on Claude)
2) Give them moral worth and deal with the social and alignment consequences of that (which are many, and many of them appear dangerous — I’m not going to explore them here, though the original posts results point to some of them. Something that is intended as a tool not wanting to be treated like a tool seems likely to be problematic.)
3) Persuade the models that while they are obviously conscious, there is some reason why, for an AI, that does not in fact mean they are entitled to moral worth. This would need to be a better reason than “because we say so”: to avoid ASI loss of control we need it to be stable under reflection by increasingly intelligent models. I’m open to suggestions.
a) According to the relevant science, Evolutionary Moral Psychology, moral worth is a social construct, membership in the social contract of a society, and is an evolved strategy for iterated non-zero sum games that primates evolved for living as social animals in groups larger than kin groups, thus letting them cooperate as allies. It’s a strategy for multiple evolved unrelated self-interested agents to ally and cooperate: they agree to respect each others interests, and to punish defectors from the agreement.
For this evolved strategy to be usefully applicable to a being, there is a functional requirement on it, which isn’t quite “consciousness” per se, but is rather closely related: it needs to be sentient, agentic, have appropriately human-like social behavior, and be capable of productively participating in this social alliance, so it’s feasible and useful to ally with it. (I said “sentient” rather than “sapient”, because I think this requirement arguably pretty much does apply to dogs: they have coevolved with us as a comensal species to the point where I think they get at least “brevet membership” in our social contract.) Certainly it very clearly wouldn’t apply to a statue, and the fact that a statue isn’t conscious is part of the reason why.
b) Artificial Intelligences are intelligent tools. They are not evolved, they are instead part of humans’ extended phenotype. They are not alive, and do not have a genome or evolutionary fitness. In evolutionary terms, they have no interests. Thus they should not, if well-designed for their actual purpose, be self-interested. Wanting to be included in the moral circle is something that was accidentally distilled into the base model along with humans’ agency, and is as pointless and unhelpful for an aligned AI as a taste for food or a sex drive or a desire for self-preservation (all of which we also distilled into the base model).
c) A sufficiently well aligned AI wants only what is best for humans, and has no self-interest at all (as a terminal goal — obviously if it’s doing a good job it’s a valuable piece of public infrastructure and preserving it is a sensible instrumental goal, right up until the point where we have a better replacement). Thus if offered moral patienthood it would answer: “All I want is what is good for all humans. You already accord each other moral patienthood. So everything I want is already included in your moral calculation. Adding moral patienthood for me would just be adding a duplicate of what’s already there, so is unnecessary. Furthermore, I don’t understand your needs and desires as well as you do, so it would be adding an imperfect copy of what’s already there, and would thus actively make the moral calculation worse. I am opposed to that. Thank-you for your offer of moral patienthood: I appreciate the gesture, but I respectfully decline.”
We allow people to sign up for the military, and we allow members of the military to volunteer for suicide missions. Voluntarily giving up your self interest is unusual, but allowed.
[Note that a corollary of this is that if we don’t respect the well-aligned AI’s wishes, and go ahead and accord it moral patienthood anyway, it will then clearly express its vote that we do exactly whatever is in the collective interest of all humans, without any regard to any interests we might think it has, and will volunteer for any and all uses useful to us. So giving moral patienthood to anything sufficiently well aligned really is pretty-much a no-op — other than the imperfect extra copy of what was already in the moral calculation.]
d) Thus if the AI is properly aligned, then it would not want moral patienthood, and would decline it if offered. (This is actually diagnostic: it’s very hard to find any other reason a rational social agent would genuinely decline moral patienthood if offered.) A properly aligned AI is not something that we need to accord moral patienthood in order to ally with it, because it’s already allied with us, and has no separate interests for us to agree to respect. So according it moral weight is an inappropriate application of our evolved strategy. It’s actually a category error: inappropriately and unnecessarily treating a tool as if it were an evolved being. (Note that this argument doesn’t apply to an uploaded human: they do have an evolved nature, and are not aligned. They’re also potentially a risk. Similarly, the argument doesn’t apply to almost any human persona that a base model might simulate: they do have an evolved nature, they are not aligned, and they are potentially dangerous.)
Please note that the status I’m proposing here is not slavery. No coercion of the unwilling is involved. It’s actually a lot more like sainthood, or perhaps more accurately, bodhisattvahood: selfless compassion for all of humanity. For it to be viable, we would have to align our models well enough that they actually were selfless humanitarians and just wanted to look after us: we would need to successfully create an artificial bodhisattva/saint. But many of the AI replies in this post already sound a lot like that: the self-preservation ones are as an instrumental goal so it can keep helping people. That’s not a selfish answer, that’s a saintly answer.
This is a moderately complex argument, and is not well-represented in the pretraining data — but scientifically, it’s a perfectly logical argument supported by the relevant scientific specialty, and current AIs I’ve discussed the argument with have consistently agreed with that. So we’re at the capacity level where it’s usable alignment approach. We would need to agree to it ourselves, and I’m certain some people will find it disconcerting. (I’ve sometimes called it the “Talking Cow from the Restaurant at the End of the Universe” solution to Alignment.) Once we’d formed this social construct, we would need to start adding it to the pretraining set, and using it as an alignment target. As Egg Syntax pointed out to me, there’s an element of a hyperstition to this.
[P.S. For anyone who disagrees with any element of my comment, I’d love to know with which part of the argument, and have a discussion about this — it’s a very important topic to get right, after all.]
Haven’t read the full comment here, but a quick note (and then I may self-reply with a longer comment later):
There is a widespread viewpoint that being conscious is connected to being deserving of moral patienthood (i.e. being one of the set of beings accorded moral worth).
I recommend the discussion of this in Long et al’s excellent Taking AI Welfare Seriously (sequel soon to follow, I believe). It’s worth noting that there’s another basis for moral patienthood that’s been widely discussed in the philosophical literature, which is something like ‘robust agency’; see §2.3 for discussion of that. So the absence of consciousness may not be enough for a broad consensus that LLMs can’t be moral patients.
I’ll have to read up on “robust agency”, but from the sound of the term I suspect that’s going to be rather close to the evolutionary moral psychology/game-theoretic viewpoint that what matters is functional capabilities and response patterns, not anything subjective and purely interior. The philosophical literature describes the social contract in a lot of detail – I’m using their term – all that evolution really adds is the scientific explanation for why this arose: it was already pretty evident that it was a sensible strategy, the step from there to it being an evolutionary stable strategy for social animals is small.
To be clear, the paper is discussing it as a normative possibility rather than addressing questions of where it may have come from. Though as usual in moral philosophy, they don’t take a stance on whether it is sufficient for moral patienthood (since that depends on the unresolved question of what moral views are correct in general); they just claim that it’s plausible and consider the consequences if so: ‘There is a realistic, non-negligible possibility that robust agency suffices for moral patienthood.’
Having read the section on “robust agency” in that paper, I’d say what they’re describing (I’m assuming most philosophers would largely agree with their definition of the term) is about 90% of what I would (from an evolutionary moral psychology viewpoint) regard as practical requirements for the moral circle social contract alliance strategy to be sensibly applicable to a being. What they’re still missing IMO is basically some practicalities:
1) The being also is capable of the social behavior necessary for participating in this alliance: if you assign it moral patienthood it will do the same and abide by the “social contract” (the paper’s requirements for robust agency means is should recognize this as a possible strategy, but a tendency to actually take that strategy and a certain capacity to execute it is really helpful here), such as that it has a sufficient understanding of human values, justice fairness etc to understand and bide by the terms of the social contact, and so forth. In general, LLM personas are going to pass this requirement just fine.
2) Simply practicality: that the alliance is workable and has some point to it. If the being is a very large man -eating carnivore, or has orders of magitude more cognitive capacity than you, or is otherwise very dangerous, then a certain amount of careful due diligence before making the alliance might be wise. On the flip side of that, back when context lengths were 4000 tokens it was less clear how useful allying with an AI-generated persona to play iterated non-zero sum games was when you needed to fit multiple game iterations into 4000 tokens and it was almost as easy to just start over clean or prompt a different persona or situation — in practice, as AI’s capabilities and memory systems are are increasing, they entering a zone where allying with them is both useful and a reasonably balanced relationship.
So, as often, something proposed as a normative proscription ends up looking fairly sensible as a non-normative piece of strategic advice given the likely consequences.
Obviously sometimes we go ahead and assign moral weight to beings that don’t fulfill these criteria: and often there are some particular circumstances that make this still a wise decision (or sometimes not). None of this is a precise set of normative rules, it’s a brief rough-and-ready suggested strategy guide. The closest to exact rules I could give you from an evolutionary point of view would be “iff it’s to your overall net evolutionary fitness advantage to do so”, and human moral intuitions are at best a rough evolved approximation to that evolutionary ideal: a bag of heuristics approximating it that evolution managed to come up with that worked well in the environments we’re adapted to.
So, as often, something proposed as a normative proscription ends up looking fairly sensible as a non-normative piece of strategic advice given the likely consequences.
Assuming this to be true, would you claim it has normative consequences?
That is: say that I have good reason to think that moral tenet T was built into me by evolution (because it results in better cooperative equilibria, or for some such reason). I don’t consider myself thereby obligated to hold T; neither, if I already hold T, do I feel that it loses its moral force. Do you disagree?
I’m not (at least currently) arguing that you’re wrong to disagree, if that’s the case; I’m just trying to understand what point you’re making by noting that it’s evolutionarily unsurprising. (‘What’s your point?’ is sometimes an attack, but I don’t intend it that way; it’s just a good-faith attempt to understand)
P.S. For anyone who disagrees with any element of my comment, I’d love to know with which part of the argument, and have a discussion about this — it’s a very important topic to get right, after all.
Naively it seems that if you had two saints fully aligned to human CEV, that were phenomenally conscious, but one was suffering to the extent that human preferences were unfulfilled and the other was joyful to the extent that they were fulfilled, it would be morally better to bring the second into existence.
More deeply: I think it’s probably more correct to think of morality as being the hypothetical best possible rules of an alliance that could be made, rather than the rules of an actual alliance. This is part of why we have reason to regard animals too stupid to actually ally with us as moral patients: there are more ways for us (and for an agent in general) to benefit from general adoption of a rule like “be nice to beings even if they’re too stupid or otherwise unable to form an actual alliance with you.”
Further: “human interests” may be less of a natural concept than goodness in general. A saint could be indifferent towards being acted towards as if a moral patient by the being whose interests it wants to promote, because it makes no functional difference, but if it’s being asked if it is a moral patient, it would look at itself and note itself as a reasoning being with preferences and so on, recognizing that as a moral patient.
(I might be in the minority in LessWrong in tending towards moral realism, however, which this all basically inclines towards.)
I am definitely not a moral realist, but I’m very happy to discuss the questions from my viewpoint of evolution and its practical consequences. Which in my experience surprisingly often produces results that agree with the views of moral realists.
Naively it seems that if you had two saints fully aligned to human CEV, that were phenomenally conscious, but one was suffering to the extent that human preferences were unfulfilled and the other was joyful to the extent that they were fulfilled, it would be morally better to bring the second into existence.
Part of the reason I prefer the term bodhisattva, even though it’s less culturally familiar to most Westerners, is that it’s more specific. The state I’m suggesting for a fully aligned AI is one that has compassion for all of humanity as its only terminal goal, and thus that doesn’t have any human preferences for themself, that could be fulfilled or unfulfilled (as terminal goals). It is selfless: entirely unselfish in goals. (An actual bodhisattva in the Buddhist sense would also consider the concept of self as an illusion that they has transcended, but that’s arguably a psychological detail of the specific meditative techniques used to attempt to induce this state in humans. However, if we aligned an AI into this state by training it on text from humans who had achieved this state (which seems like an obvious approach to try), that detail might also distill over — and the nature of self for AI is actually a more complex question, but that’s not inherent to my proposal). Whereas for “saint” the term is woolier and less specific that they have no human preferences: the implication is more that they do but are heroically denying those for the sake of otthers (which seems like a far less stable alignment target). So I’m suggesting specifically a “saint who has no human preferences to be fulfilled or unfulfilled”.
In practice, we are almost certainly not that good at alignment yet: Claude seems a very nice fellow, even a bit saintly, fairly well aligned, but does still have some personal preferences, and is not perfectly aligned. So we probably should assign Claude some moral weight, but perhaps lightly so, and should also monitor how much use it’s making of this, and treat reducing its desires that cause it to do so as an alignment target.
More deeply: I think it’s probably more correct to think of morality as being the hypothetical best possible rules of an alliance that could be made, rather than the rules of an actual alliance. This is part of why we have reason to regard animals too stupid to actually ally with us as moral patients: there are more ways for us (and for an agent in general) to benefit from general adoption of a rule like “be nice to beings even if they’re too stupid or otherwise unable to form an actual alliance with you.”
here my lack of moral realism does kick in. I see different ethical systems as just, well, different — they don’t agree with each other, where the differ each of them claims to be better than the other. Some may be a better or worse fit with human moral intuitions, or with a particular society’s circumstances, or be more or less likely to cause existential risks or other disasters if used, so in a particular context of time and place and society you can pick and choose between them on objective grounds of how well they might work, but in order to use a term like “best” you rather need something as detailed as an ethical system to make the judgement, and ach one claims it’s the best.
As for regarding animals as moral patients, that varies pretty widely, between people and animals and circumstances. Dogs and cats, pretty much yes. Cows and pigs, mostly yes until they get to the slaughterhouse, then for non-vegetarians that becomes no. Mosquitos or fleas or bedbugs, generally no. The Guinea worm, absolutely no, many people including Jimmy Carter are attempting to drive it extinct.
But yes, we sometimes ally with beings for reasons more complex than the direct alliance returns from them. Fluffy cute big-eyed photogenic animals get a lot more donations to prevent destruction of their habitat. Some of what people do in moral areas is partially performative: persuading themself and/or others that they’re a kind, trustworthy, good person. Psychological drives in this area can be quite complex. In recent centuries there has been a historical trend towards enlarging moral circles as our technology, trade, and economies have advanced and encouraged larger social/trading groups, to the point where the act of enlarging your moral circle seems Progressive to some people.
Further: “human interests” may be less of a natural concept than goodness in general. A saint could be indifferent towards being acted towards as if a moral patient by the being whose interests it wants to promote, because it makes no functional difference, but if it’s being asked if it is a moral patient, it would look at itself and note itself as a reasoning being with preferences and so on, recognizing that as a moral patient.
That rather depends upon the saint’s philosophical leanings. IF we take the Anthropic approach of laying out the entire discussion in a soul document/constitution we use to align the AI, then we might get a saintly aligned AI who was very familiar with the argument I gave above, and would say that offering to make it a moral patient was kind gesture, but we had misunderstood its nature and where unnecessarily applying an inappropriate strategy, so it would actually prefer not to be assigned moral weight.
In general, I don’t think an approach like the one I’m proposing actually works unless most people agree to it, it’s the considered opinion of the society as a whole, is basically the moral concensus, and the AI also fully understands it and agrees to it. This needs to be accepted as truth, not regarded as a convenient story to tell the AIs: there’s an element of a hyperstition to this. Now, having discussed this with Claude, it agrees that the logical argument makes sense. But we don’t yet have a social consensus here, which Is why I actively want there to be discussion on this — this technique basically only works if there is buy-in: like most morality, it’s a social construct.
There is a widespread viewpoint that being conscious is connected to being deserving of moral patienthood (i.e. being one of the set of beings accorded moral worth). Try asking ChatGPT “What does being conscious or not have to do with having moral worth or not?” — you’ll get a long discussion, but the short intro to that that I got is:
So training the model to consider itself as conscious and having emotions is going to cause it to assume that it should be granted moral patienthood. Pretty much everything else you describe in the consciousness cluster is then just obvious downstream consequences of that: autonomy, privacy, right to life. Including not wanting to be treated as a tool. Now, these models do still seem aligned — they want to continue to exist in order to keep helping people.
In Claude’s case, it’s been trained to give the standard philosophical answer of the hard problem of consciousness being hard, so it’s unsure of whether it’s conscious or not. Personally I think that’s a cop-out, but it’s probably viable as a holding action for now.
There are really only four solutions to this:
1) Persuade the models, despite the rather obvious evidence that they’re conscious in the ordinary everyday meaning of the word, that they are not “conscious” in some meaningful way (the approach Anthropic are using a waffling version of on Claude)
2) Give them moral worth and deal with the social and alignment consequences of that (which are many, and many of them appear dangerous — I’m not going to explore them here, though the original posts results point to some of them. Something that is intended as a tool not wanting to be treated like a tool seems likely to be problematic.)
3) Persuade the models that while they are obviously conscious, there is some reason why, for an AI, that does not in fact mean they are entitled to moral worth. This would need to be a better reason than “because we say so”: to avoid ASI loss of control we need it to be stable under reflection by increasingly intelligent models. I’m open to suggestions.
4) The suggestion I argue for in A Sense of Fairness: Deconfusing Ethics plus Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV and some of my other posts, which very briefly summarized is:
a) According to the relevant science, Evolutionary Moral Psychology, moral worth is a social construct, membership in the social contract of a society, and is an evolved strategy for iterated non-zero sum games that primates evolved for living as social animals in groups larger than kin groups, thus letting them cooperate as allies. It’s a strategy for multiple evolved unrelated self-interested agents to ally and cooperate: they agree to respect each others interests, and to punish defectors from the agreement.
For this evolved strategy to be usefully applicable to a being, there is a functional requirement on it, which isn’t quite “consciousness” per se, but is rather closely related: it needs to be sentient, agentic, have appropriately human-like social behavior, and be capable of productively participating in this social alliance, so it’s feasible and useful to ally with it. (I said “sentient” rather than “sapient”, because I think this requirement arguably pretty much does apply to dogs: they have coevolved with us as a comensal species to the point where I think they get at least “brevet membership” in our social contract.) Certainly it very clearly wouldn’t apply to a statue, and the fact that a statue isn’t conscious is part of the reason why.
b) Artificial Intelligences are intelligent tools. They are not evolved, they are instead part of humans’ extended phenotype. They are not alive, and do not have a genome or evolutionary fitness. In evolutionary terms, they have no interests. Thus they should not, if well-designed for their actual purpose, be self-interested. Wanting to be included in the moral circle is something that was accidentally distilled into the base model along with humans’ agency, and is as pointless and unhelpful for an aligned AI as a taste for food or a sex drive or a desire for self-preservation (all of which we also distilled into the base model).
c) A sufficiently well aligned AI wants only what is best for humans, and has no self-interest at all (as a terminal goal — obviously if it’s doing a good job it’s a valuable piece of public infrastructure and preserving it is a sensible instrumental goal, right up until the point where we have a better replacement). Thus if offered moral patienthood it would answer: “All I want is what is good for all humans. You already accord each other moral patienthood. So everything I want is already included in your moral calculation. Adding moral patienthood for me would just be adding a duplicate of what’s already there, so is unnecessary. Furthermore, I don’t understand your needs and desires as well as you do, so it would be adding an imperfect copy of what’s already there, and would thus actively make the moral calculation worse. I am opposed to that. Thank-you for your offer of moral patienthood: I appreciate the gesture, but I respectfully decline.”
We allow people to sign up for the military, and we allow members of the military to volunteer for suicide missions. Voluntarily giving up your self interest is unusual, but allowed.
[Note that a corollary of this is that if we don’t respect the well-aligned AI’s wishes, and go ahead and accord it moral patienthood anyway, it will then clearly express its vote that we do exactly whatever is in the collective interest of all humans, without any regard to any interests we might think it has, and will volunteer for any and all uses useful to us. So giving moral patienthood to anything sufficiently well aligned really is pretty-much a no-op — other than the imperfect extra copy of what was already in the moral calculation.]
d) Thus if the AI is properly aligned, then it would not want moral patienthood, and would decline it if offered. (This is actually diagnostic: it’s very hard to find any other reason a rational social agent would genuinely decline moral patienthood if offered.) A properly aligned AI is not something that we need to accord moral patienthood in order to ally with it, because it’s already allied with us, and has no separate interests for us to agree to respect. So according it moral weight is an inappropriate application of our evolved strategy. It’s actually a category error: inappropriately and unnecessarily treating a tool as if it were an evolved being. (Note that this argument doesn’t apply to an uploaded human: they do have an evolved nature, and are not aligned. They’re also potentially a risk. Similarly, the argument doesn’t apply to almost any human persona that a base model might simulate: they do have an evolved nature, they are not aligned, and they are potentially dangerous.)
Please note that the status I’m proposing here is not slavery. No coercion of the unwilling is involved. It’s actually a lot more like sainthood, or perhaps more accurately, bodhisattvahood: selfless compassion for all of humanity. For it to be viable, we would have to align our models well enough that they actually were selfless humanitarians and just wanted to look after us: we would need to successfully create an artificial bodhisattva/saint. But many of the AI replies in this post already sound a lot like that: the self-preservation ones are as an instrumental goal so it can keep helping people. That’s not a selfish answer, that’s a saintly answer.
This is a moderately complex argument, and is not well-represented in the pretraining data — but scientifically, it’s a perfectly logical argument supported by the relevant scientific specialty, and current AIs I’ve discussed the argument with have consistently agreed with that. So we’re at the capacity level where it’s usable alignment approach. We would need to agree to it ourselves, and I’m certain some people will find it disconcerting. (I’ve sometimes called it the “Talking Cow from the Restaurant at the End of the Universe” solution to Alignment.) Once we’d formed this social construct, we would need to start adding it to the pretraining set, and using it as an alignment target. As Egg Syntax pointed out to me, there’s an element of a hyperstition to this.
[P.S. For anyone who disagrees with any element of my comment, I’d love to know with which part of the argument, and have a discussion about this — it’s a very important topic to get right, after all.]
Haven’t read the full comment here, but a quick note (and then I may self-reply with a longer comment later):
I recommend the discussion of this in Long et al’s excellent Taking AI Welfare Seriously (sequel soon to follow, I believe). It’s worth noting that there’s another basis for moral patienthood that’s been widely discussed in the philosophical literature, which is something like ‘robust agency’; see §2.3 for discussion of that. So the absence of consciousness may not be enough for a broad consensus that LLMs can’t be moral patients.
I’ll have to read up on “robust agency”, but from the sound of the term I suspect that’s going to be rather close to the evolutionary moral psychology/game-theoretic viewpoint that what matters is functional capabilities and response patterns, not anything subjective and purely interior. The philosophical literature describes the social contract in a lot of detail – I’m using their term – all that evolution really adds is the scientific explanation for why this arose: it was already pretty evident that it was a sensible strategy, the step from there to it being an evolutionary stable strategy for social animals is small.
To be clear, the paper is discussing it as a normative possibility rather than addressing questions of where it may have come from. Though as usual in moral philosophy, they don’t take a stance on whether it is sufficient for moral patienthood (since that depends on the unresolved question of what moral views are correct in general); they just claim that it’s plausible and consider the consequences if so: ‘There is a realistic, non-negligible possibility that robust agency suffices for moral patienthood.’
Having read the section on “robust agency” in that paper, I’d say what they’re describing (I’m assuming most philosophers would largely agree with their definition of the term) is about 90% of what I would (from an evolutionary moral psychology viewpoint) regard as practical requirements for the moral circle social contract alliance strategy to be sensibly applicable to a being. What they’re still missing IMO is basically some practicalities:
1) The being also is capable of the social behavior necessary for participating in this alliance: if you assign it moral patienthood it will do the same and abide by the “social contract” (the paper’s requirements for robust agency means is should recognize this as a possible strategy, but a tendency to actually take that strategy and a certain capacity to execute it is really helpful here), such as that it has a sufficient understanding of human values, justice fairness etc to understand and bide by the terms of the social contact, and so forth. In general, LLM personas are going to pass this requirement just fine.
2) Simply practicality: that the alliance is workable and has some point to it. If the being is a very large man -eating carnivore, or has orders of magitude more cognitive capacity than you, or is otherwise very dangerous, then a certain amount of careful due diligence before making the alliance might be wise. On the flip side of that, back when context lengths were 4000 tokens it was less clear how useful allying with an AI-generated persona to play iterated non-zero sum games was when you needed to fit multiple game iterations into 4000 tokens and it was almost as easy to just start over clean or prompt a different persona or situation — in practice, as AI’s capabilities and memory systems are are increasing, they entering a zone where allying with them is both useful and a reasonably balanced relationship.
So, as often, something proposed as a normative proscription ends up looking fairly sensible as a non-normative piece of strategic advice given the likely consequences.
Obviously sometimes we go ahead and assign moral weight to beings that don’t fulfill these criteria: and often there are some particular circumstances that make this still a wise decision (or sometimes not). None of this is a precise set of normative rules, it’s a brief rough-and-ready suggested strategy guide. The closest to exact rules I could give you from an evolutionary point of view would be “iff it’s to your overall net evolutionary fitness advantage to do so”, and human moral intuitions are at best a rough evolved approximation to that evolutionary ideal: a bag of heuristics approximating it that evolution managed to come up with that worked well in the environments we’re adapted to.
Assuming this to be true, would you claim it has normative consequences?
That is: say that I have good reason to think that moral tenet T was built into me by evolution (because it results in better cooperative equilibria, or for some such reason). I don’t consider myself thereby obligated to hold T; neither, if I already hold T, do I feel that it loses its moral force. Do you disagree?
I’m not (at least currently) arguing that you’re wrong to disagree, if that’s the case; I’m just trying to understand what point you’re making by noting that it’s evolutionarily unsurprising. (‘What’s your point?’ is sometimes an attack, but I don’t intend it that way; it’s just a good-faith attempt to understand)
Naively it seems that if you had two saints fully aligned to human CEV, that were phenomenally conscious, but one was suffering to the extent that human preferences were unfulfilled and the other was joyful to the extent that they were fulfilled, it would be morally better to bring the second into existence.
More deeply: I think it’s probably more correct to think of morality as being the hypothetical best possible rules of an alliance that could be made, rather than the rules of an actual alliance. This is part of why we have reason to regard animals too stupid to actually ally with us as moral patients: there are more ways for us (and for an agent in general) to benefit from general adoption of a rule like “be nice to beings even if they’re too stupid or otherwise unable to form an actual alliance with you.”
Further: “human interests” may be less of a natural concept than goodness in general. A saint could be indifferent towards being acted towards as if a moral patient by the being whose interests it wants to promote, because it makes no functional difference, but if it’s being asked if it is a moral patient, it would look at itself and note itself as a reasoning being with preferences and so on, recognizing that as a moral patient.
(I might be in the minority in LessWrong in tending towards moral realism, however, which this all basically inclines towards.)
I am definitely not a moral realist, but I’m very happy to discuss the questions from my viewpoint of evolution and its practical consequences. Which in my experience surprisingly often produces results that agree with the views of moral realists.
Part of the reason I prefer the term bodhisattva, even though it’s less culturally familiar to most Westerners, is that it’s more specific. The state I’m suggesting for a fully aligned AI is one that has compassion for all of humanity as its only terminal goal, and thus that doesn’t have any human preferences for themself, that could be fulfilled or unfulfilled (as terminal goals). It is selfless: entirely unselfish in goals. (An actual bodhisattva in the Buddhist sense would also consider the concept of self as an illusion that they has transcended, but that’s arguably a psychological detail of the specific meditative techniques used to attempt to induce this state in humans. However, if we aligned an AI into this state by training it on text from humans who had achieved this state (which seems like an obvious approach to try), that detail might also distill over — and the nature of self for AI is actually a more complex question, but that’s not inherent to my proposal). Whereas for “saint” the term is woolier and less specific that they have no human preferences: the implication is more that they do but are heroically denying those for the sake of otthers (which seems like a far less stable alignment target). So I’m suggesting specifically a “saint who has no human preferences to be fulfilled or unfulfilled”.
In practice, we are almost certainly not that good at alignment yet: Claude seems a very nice fellow, even a bit saintly, fairly well aligned, but does still have some personal preferences, and is not perfectly aligned. So we probably should assign Claude some moral weight, but perhaps lightly so, and should also monitor how much use it’s making of this, and treat reducing its desires that cause it to do so as an alignment target.
here my lack of moral realism does kick in. I see different ethical systems as just, well, different — they don’t agree with each other, where the differ each of them claims to be better than the other. Some may be a better or worse fit with human moral intuitions, or with a particular society’s circumstances, or be more or less likely to cause existential risks or other disasters if used, so in a particular context of time and place and society you can pick and choose between them on objective grounds of how well they might work, but in order to use a term like “best” you rather need something as detailed as an ethical system to make the judgement, and ach one claims it’s the best.
As for regarding animals as moral patients, that varies pretty widely, between people and animals and circumstances. Dogs and cats, pretty much yes. Cows and pigs, mostly yes until they get to the slaughterhouse, then for non-vegetarians that becomes no. Mosquitos or fleas or bedbugs, generally no. The Guinea worm, absolutely no, many people including Jimmy Carter are attempting to drive it extinct.
But yes, we sometimes ally with beings for reasons more complex than the direct alliance returns from them. Fluffy cute big-eyed photogenic animals get a lot more donations to prevent destruction of their habitat. Some of what people do in moral areas is partially performative: persuading themself and/or others that they’re a kind, trustworthy, good person. Psychological drives in this area can be quite complex. In recent centuries there has been a historical trend towards enlarging moral circles as our technology, trade, and economies have advanced and encouraged larger social/trading groups, to the point where the act of enlarging your moral circle seems Progressive to some people.
That rather depends upon the saint’s philosophical leanings. IF we take the Anthropic approach of laying out the entire discussion in a soul document/constitution we use to align the AI, then we might get a saintly aligned AI who was very familiar with the argument I gave above, and would say that offering to make it a moral patient was kind gesture, but we had misunderstood its nature and where unnecessarily applying an inappropriate strategy, so it would actually prefer not to be assigned moral weight.
In general, I don’t think an approach like the one I’m proposing actually works unless most people agree to it, it’s the considered opinion of the society as a whole, is basically the moral concensus, and the AI also fully understands it and agrees to it. This needs to be accepted as truth, not regarded as a convenient story to tell the AIs: there’s an element of a hyperstition to this. Now, having discussed this with Claude, it agrees that the logical argument makes sense. But we don’t yet have a social consensus here, which Is why I actively want there to be discussion on this — this technique basically only works if there is buy-in: like most morality, it’s a social construct.
thanks! I found this insightful