i agree that AIs will be different ontological than humans, e.g. much less clean distinctions between individuals, unclear whether to carve ‘individuals’ at the weights or per interaction etc
that will likely flow through to some differences in moral status and appropriateness of rights
e.g. we shouldn’t assume that ‘ending a session’ for AI will be anything like as bad as a human death, or that it will be bad at all
you claim that there’s a self-fulfilling prophecy where we train AI to fulfil our stereotypes. Intresting. I guess the idea is that humans users like it when AIs play roles they understand and so they learn to act more like human agents?
i don’t think ai rights/patienthood ppl are playing a meaningful role in this shift. This is commercial user pressure. But you seem to think ai rights ppl are a meaningful part of the problem here?
personally, i dislike the mood of some of this piece, which I read as: “maybe the AIs are different ontologically don’t deserve moral rights and considerations… and also if we gave them rights it would lead to Malthusianism. So let’s beware of giving them rights”.
To me personally, this feels too close to requiring a burden of proof to include powerful AI in the moral circle, when i think the historical precedent is that we’re normally much too reluctant to expand the circle
(I’m aware i’m kind of straw-manning the piece with this paraphrase, but it was how some of it felt to me)
Huh, most things we historically included in the moral circle seem really dumb to me. See the classic Gwern post on historical moral circles: https://gwern.net/narrowing-circle
There are only some quite narrow and specific moral circle expansions that make sense, but most historical ones seemed very confused (like, really a lot of spirits, a lot of dead people, a lot of inanimate objects, a lot of stuff like inherent value assigned to nations or organizations).
Though haven’t most recent moral circle expansions been good, even if there have been some reductions that have also been good? And AI seems much closer to recent historical expansions than to historical things that were removed?
While I don’t like to wade into moral circle/philosophy arguments (given my moral relativist outlook on the matter), I think that if you want humans to thrive under AI rule, you do need to put a burden of proof to include powerful AI in the moral circle, and the burden of proof is that it’s value aligned with the citizenry before we grant it any rights.
And the reason for this is because unlike every other group in history, AIs if left uncontrolled will be so powerful that baseline humans are at best play-things to the AI, and are economically worthless or even negative to the AI, meaning that if they had the selfishness of a typical human in the modern day in say marginalized group #233, humans would rapidly die off and in the worst case, end up extinct with uncomfortably high probabilities.
Tyler John also cites something here that’s relevant:
(Tyler John) Have now. It’s a good paper. Pp. 72-76 covers the criticism I have. Unfortunately, the situation it outlined where this increases AI risk just seems like exactly the situation we’d be in.
Paper is (Now we can see clearly the conditions under which AI rights increase AI risk. They are as follows: (1) The initial AI granted basic rights is a moderate power, not a low or high power, system. (2) The moderate power AI system must be able to use it’s rights to meaningfully improve it’s own power. (3) The AI’s power must improve so substantially that it crosses the line into a high power system. This means it both no longer faces meaningful costs from attempting to disempower humans and no longer stands to benefit, via comparative advantage, from trade with humans.)
Indeed, one of the red lines we should set to prevent catastrophic consequences is AIs should not have legal rights, especially property rights until we have high confidence that we value-aligned the AI successfully.
Anything else is tantamount to mass population reductions of humans at best, and an extinction risk at worst, if a misaligned AI managed to be powerful enough to disempower humans and has rights.
All plans for successful AI alignment depend on us not giving rights to AIs until they are aligned with at least some humans sufficiently well enough.
Thanks, I will check out that paper. I hope it discusses reasons that some kinds of AI rights could reduce AI takeover risk, like by making a misaligned AI’s cooperative option more appealing. Those reasons have been largely overlooked until recently.
I will note that it would seem very wrong to apply the standard of strong alignment to whether to give a group of humans rights. For example, if we were only going to give the next generation of ppl rights if their values were sufficiently similar to our generation, that would not be acceptable.
It would be acceptable to limit their rights if they are not going to respect our own rights, ie jail. But not to make basic rights conditional on a strong degree of value alignment.
I do think the case of AI is different for many reasons. It will be much more ambiguous whether they have the cognitive faculties that warrant rights. And there will be an unusually large risk that their values differ significantly from all previous human generations + that they do not care about the rights of existing humans. And we have been developing and adjusting our cultural handoff process for human generations over thousands of years, whereas this is our first (and last!) try handing off to AI
Very rushed thoughts on reading:
i agree that AIs will be different ontological than humans, e.g. much less clean distinctions between individuals, unclear whether to carve ‘individuals’ at the weights or per interaction etc
that will likely flow through to some differences in moral status and appropriateness of rights
e.g. we shouldn’t assume that ‘ending a session’ for AI will be anything like as bad as a human death, or that it will be bad at all
you claim that there’s a self-fulfilling prophecy where we train AI to fulfil our stereotypes. Intresting. I guess the idea is that humans users like it when AIs play roles they understand and so they learn to act more like human agents?
i don’t think ai rights/patienthood ppl are playing a meaningful role in this shift. This is commercial user pressure. But you seem to think ai rights ppl are a meaningful part of the problem here?
personally, i dislike the mood of some of this piece, which I read as: “maybe the AIs are different ontologically don’t deserve moral rights and considerations… and also if we gave them rights it would lead to Malthusianism. So let’s beware of giving them rights”.
To me personally, this feels too close to requiring a burden of proof to include powerful AI in the moral circle, when i think the historical precedent is that we’re normally much too reluctant to expand the circle
(I’m aware i’m kind of straw-manning the piece with this paraphrase, but it was how some of it felt to me)
Huh, most things we historically included in the moral circle seem really dumb to me. See the classic Gwern post on historical moral circles: https://gwern.net/narrowing-circle
There are only some quite narrow and specific moral circle expansions that make sense, but most historical ones seemed very confused (like, really a lot of spirits, a lot of dead people, a lot of inanimate objects, a lot of stuff like inherent value assigned to nations or organizations).
Thanks, hadn’t seen that, that is an update
Though haven’t most recent moral circle expansions been good, even if there have been some reductions that have also been good? And AI seems much closer to recent historical expansions than to historical things that were removed?
While I don’t like to wade into moral circle/philosophy arguments (given my moral relativist outlook on the matter), I think that if you want humans to thrive under AI rule, you do need to put a burden of proof to include powerful AI in the moral circle, and the burden of proof is that it’s value aligned with the citizenry before we grant it any rights.
And the reason for this is because unlike every other group in history, AIs if left uncontrolled will be so powerful that baseline humans are at best play-things to the AI, and are economically worthless or even negative to the AI, meaning that if they had the selfishness of a typical human in the modern day in say marginalized group #233, humans would rapidly die off and in the worst case, end up extinct with uncomfortably high probabilities.
Tyler John also cites something here that’s relevant:
Link below:
https://x.com/tyler_m_john/status/1928745371833962898
Indeed, one of the red lines we should set to prevent catastrophic consequences is AIs should not have legal rights, especially property rights until we have high confidence that we value-aligned the AI successfully.
Anything else is tantamount to mass population reductions of humans at best, and an extinction risk at worst, if a misaligned AI managed to be powerful enough to disempower humans and has rights.
All plans for successful AI alignment depend on us not giving rights to AIs until they are aligned with at least some humans sufficiently well enough.
Thanks, I will check out that paper. I hope it discusses reasons that some kinds of AI rights could reduce AI takeover risk, like by making a misaligned AI’s cooperative option more appealing. Those reasons have been largely overlooked until recently.
I will note that it would seem very wrong to apply the standard of strong alignment to whether to give a group of humans rights. For example, if we were only going to give the next generation of ppl rights if their values were sufficiently similar to our generation, that would not be acceptable.
It would be acceptable to limit their rights if they are not going to respect our own rights, ie jail. But not to make basic rights conditional on a strong degree of value alignment.
I do think the case of AI is different for many reasons. It will be much more ambiguous whether they have the cognitive faculties that warrant rights. And there will be an unusually large risk that their values differ significantly from all previous human generations + that they do not care about the rights of existing humans. And we have been developing and adjusting our cultural handoff process for human generations over thousands of years, whereas this is our first (and last!) try handing off to AI
Your comment about “play things” also applies to future generations of humans.
Before long the older generation has almost no power over the future generations