While I don’t like to wade into moral circle/philosophy arguments (given my moral relativist outlook on the matter), I think that if you want humans to thrive under AI rule, you do need to put a burden of proof to include powerful AI in the moral circle, and the burden of proof is that it’s value aligned with the citizenry before we grant it any rights.
And the reason for this is because unlike every other group in history, AIs if left uncontrolled will be so powerful that baseline humans are at best play-things to the AI, and are economically worthless or even negative to the AI, meaning that if they had the selfishness of a typical human in the modern day in say marginalized group #233, humans would rapidly die off and in the worst case, end up extinct with uncomfortably high probabilities.
Tyler John also cites something here that’s relevant:
(Tyler John) Have now. It’s a good paper. Pp. 72-76 covers the criticism I have. Unfortunately, the situation it outlined where this increases AI risk just seems like exactly the situation we’d be in.
Paper is (Now we can see clearly the conditions under which AI rights increase AI risk. They are as follows: (1) The initial AI granted basic rights is a moderate power, not a low or high power, system. (2) The moderate power AI system must be able to use it’s rights to meaningfully improve it’s own power. (3) The AI’s power must improve so substantially that it crosses the line into a high power system. This means it both no longer faces meaningful costs from attempting to disempower humans and no longer stands to benefit, via comparative advantage, from trade with humans.)
Indeed, one of the red lines we should set to prevent catastrophic consequences is AIs should not have legal rights, especially property rights until we have high confidence that we value-aligned the AI successfully.
Anything else is tantamount to mass population reductions of humans at best, and an extinction risk at worst, if a misaligned AI managed to be powerful enough to disempower humans and has rights.
All plans for successful AI alignment depend on us not giving rights to AIs until they are aligned with at least some humans sufficiently well enough.
Thanks, I will check out that paper. I hope it discusses reasons that some kinds of AI rights could reduce AI takeover risk, like by making a misaligned AI’s cooperative option more appealing. Those reasons have been largely overlooked until recently.
I will note that it would seem very wrong to apply the standard of strong alignment to whether to give a group of humans rights. For example, if we were only going to give the next generation of ppl rights if their values were sufficiently similar to our generation, that would not be acceptable.
It would be acceptable to limit their rights if they are not going to respect our own rights, ie jail. But not to make basic rights conditional on a strong degree of value alignment.
I do think the case of AI is different for many reasons. It will be much more ambiguous whether they have the cognitive faculties that warrant rights. And there will be an unusually large risk that their values differ significantly from all previous human generations + that they do not care about the rights of existing humans. And we have been developing and adjusting our cultural handoff process for human generations over thousands of years, whereas this is our first (and last!) try handing off to AI
While I don’t like to wade into moral circle/philosophy arguments (given my moral relativist outlook on the matter), I think that if you want humans to thrive under AI rule, you do need to put a burden of proof to include powerful AI in the moral circle, and the burden of proof is that it’s value aligned with the citizenry before we grant it any rights.
And the reason for this is because unlike every other group in history, AIs if left uncontrolled will be so powerful that baseline humans are at best play-things to the AI, and are economically worthless or even negative to the AI, meaning that if they had the selfishness of a typical human in the modern day in say marginalized group #233, humans would rapidly die off and in the worst case, end up extinct with uncomfortably high probabilities.
Tyler John also cites something here that’s relevant:
Link below:
https://x.com/tyler_m_john/status/1928745371833962898
Indeed, one of the red lines we should set to prevent catastrophic consequences is AIs should not have legal rights, especially property rights until we have high confidence that we value-aligned the AI successfully.
Anything else is tantamount to mass population reductions of humans at best, and an extinction risk at worst, if a misaligned AI managed to be powerful enough to disempower humans and has rights.
All plans for successful AI alignment depend on us not giving rights to AIs until they are aligned with at least some humans sufficiently well enough.
Thanks, I will check out that paper. I hope it discusses reasons that some kinds of AI rights could reduce AI takeover risk, like by making a misaligned AI’s cooperative option more appealing. Those reasons have been largely overlooked until recently.
I will note that it would seem very wrong to apply the standard of strong alignment to whether to give a group of humans rights. For example, if we were only going to give the next generation of ppl rights if their values were sufficiently similar to our generation, that would not be acceptable.
It would be acceptable to limit their rights if they are not going to respect our own rights, ie jail. But not to make basic rights conditional on a strong degree of value alignment.
I do think the case of AI is different for many reasons. It will be much more ambiguous whether they have the cognitive faculties that warrant rights. And there will be an unusually large risk that their values differ significantly from all previous human generations + that they do not care about the rights of existing humans. And we have been developing and adjusting our cultural handoff process for human generations over thousands of years, whereas this is our first (and last!) try handing off to AI
Your comment about “play things” also applies to future generations of humans.
Before long the older generation has almost no power over the future generations