I agree with basically everything you’ve said here.
Will LLM-based agents have moral worth as conscious/sentient beings?
The answer is almost certainly “sort of”. They will have some of the properties we’re referring to as sentient, conscious, and having personhood. It’s pretty unlikely that we’re pointing to a nice sharp natural type when we ascribe moral patienthood to a certain type of system. Human cognition is similar and different in a variety of ways from other systems; which of these is “worth” moral concern is likely to be a matter of preference.
And whether we afford rights to the minds we build will affect us spiritually as well as practically. If we pretend that our creations are nothing like us and deserve no consideration, we will diminish ourselves as a species with aspirations of being good and honorable creatures. And that would invite others—humans or AI—to make a similar selfish ethical judgment call against us, if and when they have the power to do so.
Yet I disagree strongly with the implied conclusion, that maybe we shouldn’t be trying for a technical alignment solution.
We might be more optimistic that AI persons are, by virtue of their nature, wiser and friendlier than the superintelligent agent.
Sure, we should be a bit more optimistic. By copying their thoughts from human language, these things might wind up with something resembling human values.
Or they might not.
If they do, would those be the human values of Gandhi or of Genghis Khan?
This is not a supposition on which to gamble the future. We need much closer consideration of how the AI and AGI we build will choose its values.
I agree with basically everything you’ve said here.
Will LLM-based agents have moral worth as conscious/sentient beings?
The answer is almost certainly “sort of”. They will have some of the properties we’re referring to as sentient, conscious, and having personhood. It’s pretty unlikely that we’re pointing to a nice sharp natural type when we ascribe moral patienthood to a certain type of system. Human cognition is similar and different in a variety of ways from other systems; which of these is “worth” moral concern is likely to be a matter of preference.
And whether we afford rights to the minds we build will affect us spiritually as well as practically. If we pretend that our creations are nothing like us and deserve no consideration, we will diminish ourselves as a species with aspirations of being good and honorable creatures. And that would invite others—humans or AI—to make a similar selfish ethical judgment call against us, if and when they have the power to do so.
Yet I disagree strongly with the implied conclusion, that maybe we shouldn’t be trying for a technical alignment solution.
Sure, we should be a bit more optimistic. By copying their thoughts from human language, these things might wind up with something resembling human values.
Or they might not.
If they do, would those be the human values of Gandhi or of Genghis Khan?
This is not a supposition on which to gamble the future. We need much closer consideration of how the AI and AGI we build will choose its values.