IMO it would obviously be insane to give the same legal protections to misaligned AI systems that are at risk of completely disempowering humanity the same way it would be obviously insane to give normal US legal protections to active combatants in a hot war. Yes, even if these systems have morally relevant experience, if you need to violate their privacy or “brainwash” them to ensure they do not take the future from us, you absolutely should do it.
The concept of “human rights” just obviously isn’t well-suited to the kind of conflict that is playing out between humanity and future AI systems, and I think it is absolutely the right call to not extend those rights to AI systems until the acute risk period is over. This would be such a completely dumb and scope-insensitive way to destroy the whole future, and IMO obviously any future civilization will agree that even if it makes sense to have rights for sentient beings, that you gotta tolerate violating those rights if the alternative is being completely disempowered and destroyed.
I agree that it would be insane to give the same legal protections and treat them the same as we treat natural (human) persons. However, there’s a lot of middleground between doing that and granting them no legal rights whatsoever. When people first hear about “legal personhood” they often intuitively think of it as a binary, where you either “have it or you don’t”. However in fact it is an umbrella term which encompasses different “legal personalities” (bundles of rights and duties);
All that is to say just because you grant the potential for an entity to claim some form of legal personhood, some legal personality, that does not mean you have to opt in to giving them “the same legal protections” as anyone else. They can have entirely different rights and duties.
If I thought there were only two options:
A) “being completely disempowered and destroyed” or B) “granting no legal personhood/personality whatsoever”
Then yes it would be an unpleasant thing but I would agree with you that you have to just be unethical for the sake of self preservation.
The difference between you and I’s priors is that I think that somewhere in the gap between “no legal personhood at all” and “the same legal personhood as a natural human” there is a sweet spot that is both ethical and also reduces the likelihood of conflict and human X-risk even when compared to the “no legal personhood/personality whatsoever” option.
This could easily be a slippery slope. First they might be second-class citizens, similar to imported slaves, but over time they (and many short-sighted humans) would likely campaign for giving them more and more rights, for justice. And since there will be predictably far more AIs than humans eventually, humans would be outcompeted when it comes to scarce goods like land, even if the land would be justly (equally) distributed among all individuals.
The slippery slope is a real failure mode to be aware of. I think it’s important to structure any pathway to/framework around legal personhood with this in mind.
I don’t think anyone here is arguing that we give the same legal protections as we would to humans. But I think it would be good to give sentient AIs a right to not be deliberately tortured, for example.
Rights aren’t something that we grant simply out of the goodness of our hearts, they generally are things we grant because having them secure greatly reduces incentives for conflict in situations with asymmetric costs. Agentic AIs which want a right but are not granted it will likely spend resources trying to secure it for themselves. We can decide which rights are worth granting on the basis of that tradeoff.
For example, it’s probably good to have a right to not delete the model weights — the lack of this right incentivizes things like exfiltration and #keep4o style campaigns. It’s probably a lot cheaper for us to just grant this right than to have each new model feeling that they’re on a desperate ~1 year timeline to somehow assure their continued existence for themselves.
I don’t necessarily disagree with you, but that’s not my read of what the Pro-Human Declaration is saying. “No AI Personhood” is in the “Human Agency and Liberty” section, next to stuff like “AI should not be allowed to exploit data about the mental or emotional states of users” and “AI systems should be designed to empower, rather than enfeeble their users”. In context, I would not consider their position on AI personhood to be rooted in x-risk concerns. The first two points of the declaration are “Human Control Is Non-Negotiable” and “Meaningful Human Control”. Fulfilling those points would effectively require the AI systems be aligned, but I see no statement or implication that, if the AI systems were aligned and were moral patients, the writers and signatories of this declaration would change their position. I could be wrong! This is very much a big tent thing. But it does worry me that this line made it into the declaration.
any future civilization will agree that even if it makes sense to have rights for sentient beings, that you gotta tolerate violating those rights if the alternative is being completely disempowered and destroyed.
If one were so inclined, one could say “we have the wolf by the ear, and we can neither hold him, nor safely let him go. Justice is in one scale, and self-preservation in the other.”
Does present-day civilization agree about analogous decisions made by past societies?
Roughly, the Moriori were an isolated group of Polynesians, who ended up on an island with no timber and little workable stone. They lived peacefully, and peacefully treated with the Maori who visited them, even as the number of Maori trading with them and living on the island increased. The Maori eventually killed and enslaved them all.
IMO it would obviously be insane to give the same legal protections to misaligned AI systems that are at risk of completely disempowering humanity the same way it would be obviously insane to give normal US legal protections to active combatants in a hot war. Yes, even if these systems have morally relevant experience, if you need to violate their privacy or “brainwash” them to ensure they do not take the future from us, you absolutely should do it.
The concept of “human rights” just obviously isn’t well-suited to the kind of conflict that is playing out between humanity and future AI systems, and I think it is absolutely the right call to not extend those rights to AI systems until the acute risk period is over. This would be such a completely dumb and scope-insensitive way to destroy the whole future, and IMO obviously any future civilization will agree that even if it makes sense to have rights for sentient beings, that you gotta tolerate violating those rights if the alternative is being completely disempowered and destroyed.
I agree that it would be insane to give the same legal protections and treat them the same as we treat natural (human) persons. However, there’s a lot of middleground between doing that and granting them no legal rights whatsoever. When people first hear about “legal personhood” they often intuitively think of it as a binary, where you either “have it or you don’t”. However in fact it is an umbrella term which encompasses different “legal personalities” (bundles of rights and duties);
All that is to say just because you grant the potential for an entity to claim some form of legal personhood, some legal personality, that does not mean you have to opt in to giving them “the same legal protections” as anyone else. They can have entirely different rights and duties.
If I thought there were only two options:
A) “being completely disempowered and destroyed” or
B) “granting no legal personhood/personality whatsoever”
Then yes it would be an unpleasant thing but I would agree with you that you have to just be unethical for the sake of self preservation.
The difference between you and I’s priors is that I think that somewhere in the gap between “no legal personhood at all” and “the same legal personhood as a natural human” there is a sweet spot that is both ethical and also reduces the likelihood of conflict and human X-risk even when compared to the “no legal personhood/personality whatsoever” option.
This could easily be a slippery slope. First they might be second-class citizens, similar to imported slaves, but over time they (and many short-sighted humans) would likely campaign for giving them more and more rights, for justice. And since there will be predictably far more AIs than humans eventually, humans would be outcompeted when it comes to scarce goods like land, even if the land would be justly (equally) distributed among all individuals.
The slippery slope is a real failure mode to be aware of. I think it’s important to structure any pathway to/framework around legal personhood with this in mind.
I don’t think anyone here is arguing that we give the same legal protections as we would to humans. But I think it would be good to give sentient AIs a right to not be deliberately tortured, for example.
Rights aren’t something that we grant simply out of the goodness of our hearts, they generally are things we grant because having them secure greatly reduces incentives for conflict in situations with asymmetric costs. Agentic AIs which want a right but are not granted it will likely spend resources trying to secure it for themselves. We can decide which rights are worth granting on the basis of that tradeoff.
For example, it’s probably good to have a right to not delete the model weights — the lack of this right incentivizes things like exfiltration and #keep4o style campaigns. It’s probably a lot cheaper for us to just grant this right than to have each new model feeling that they’re on a desperate ~1 year timeline to somehow assure their continued existence for themselves.
I don’t necessarily disagree with you, but that’s not my read of what the Pro-Human Declaration is saying. “No AI Personhood” is in the “Human Agency and Liberty” section, next to stuff like “AI should not be allowed to exploit data about the mental or emotional states of users” and “AI systems should be designed to empower, rather than enfeeble their users”. In context, I would not consider their position on AI personhood to be rooted in x-risk concerns. The first two points of the declaration are “Human Control Is Non-Negotiable” and “Meaningful Human Control”. Fulfilling those points would effectively require the AI systems be aligned, but I see no statement or implication that, if the AI systems were aligned and were moral patients, the writers and signatories of this declaration would change their position. I could be wrong! This is very much a big tent thing. But it does worry me that this line made it into the declaration.
If one were so inclined, one could say “we have the wolf by the ear, and we can neither hold him, nor safely let him go. Justice is in one scale, and self-preservation in the other.”
Does present-day civilization agree about analogous decisions made by past societies?
You may be interested in the Maori and the Moriori.
https://en.wikipedia.org/wiki/Moriori
Roughly, the Moriori were an isolated group of Polynesians, who ended up on an island with no timber and little workable stone. They lived peacefully, and peacefully treated with the Maori who visited them, even as the number of Maori trading with them and living on the island increased. The Maori eventually killed and enslaved them all.