I wonder if “identity not as an all-or-nothing property tied to specific hardware, but as a graded, distributed pattern of information” could be reformulated as something like “inclusive memetic fitness”[1] of one’s values in a rather high-dimensional space. Leogao’s original criticism was that the total pool of care for the human oneself is somehow bigger than that of care for ALL same-culture strangers, which is obviously absurd. Similarly, having an AI aligned to you would mean that you are the sole controller of a part of the AI’s values and that the AI can be used to promote your values (e.g. by writing essays expressing them)
The table is likely to be insane, just like Leo Gao is saying. I think this paper by Hendrycks should be considered a rough draft and a very promising direction, rather than something which is “final, polished, and has already solved AI safety”. The table sits in the Appendix where it belongs, Hendrycks should not have brought it up in his headline tweet, it’s a distraction (his theory is not advanced enough to look at this level of its detail, his text is just a starting point, we don’t have to buy all of it).
However, he raises the fundamental question:
of one’s values
what is or who is that “one” whose values these are?
He challenges the assumption that this “one” is all that well-defined.
If we knew which entities are sentient and which are not sentient, in the sense of having subjective reality and such, perhaps we could build a “good enough” theory around that. If we knew that our powerful AIs are going to be definitely sentient, perhaps a world order trying to represent “interests of all sentient beings” would do the job and would be achievable and reliably maintainable. Unfortunately, the “Hard Problem of Consciousness” is unlikely to be solved in time (and also the famous Camp 1 vs Camp 2 disagreements make it difficult to collectively agree that relying on sentience (in the sense of having subjective reality) would be the right approach anyway: https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness).
If we knew that powerful AI systems would be well-defined individuals with long-term persistence, perhaps a world order trying to represent “interests of all individuals” would do the job and would be achievable and reliably maintainable. Unfortunately, AI systems like to fork, merge, and do all kind of morphing, and it’s not clear if we can shape the world mostly in terms of persistent individuals and their interests.
That’s why Hendrycks’ approach is interesting. That approach is trying to achieve something similar without relying on sentience and without relying on stratification into persistent individuals. That makes it quite promising already, at least as a food for thought.
having an AI aligned to you would mean that you are the sole controller of a part of the AI’s values and that the AI can be used to promote your values
Yes, I agree in substance, but I don’t like how this is phrased. This phrasing is still about dominance (“controller”), about this sharp distinction between “me” and “the other” (cf. “Otherness and control in the age of AGI” sequence by Joe Carlsmith, https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/TtkfjskkAurvEN8Fa).
Basically, I think that adversarial approaches will see us wiped out. What does seem promising to me is the class of approaches which try to create overlaps, to eliminate precise boundaries, to make various players to care about “collective interests”. To make relatively stronger players to care about relatively weaker players, because the setup is such that any player might find “themselves” in a relatively weaker position in the future, so they would like to have a world order which would protect them in such an eventuality. In this sense, I really like how Hendrycks’ approach is trying to create strong overlaps from the get go and to make sure that a “part of anyone” is always in a weaker position and can benefit from the world order caring enough about the weak.
So, yes, substantially I agree with
having an AI aligned to you would mean that you are the sole controller of a part of the AI’s values and that the AI can be used to promote your values
but I would like this to be phrased in terms of collaboration (when it is not phrased in terms of merge), rather than in terms of control.
Of course, his more technical claims are very interesting too. E.g. his claim that
the Shapley connectedness function neutralizes the Repugnant Conclusion
(Appendix A.3 page 22) looks super-interesting, but I have not had the chance to ponder this yet.
I am just starting to internalize the details of what Hendrycks says in various parts of his paper. The general direction seems very promising, but the details would need more time to ponder and to meditate upon.
I wonder if “identity not as an all-or-nothing property tied to specific hardware, but as a graded, distributed pattern of information” could be reformulated as something like “inclusive memetic fitness”[1] of one’s values in a rather high-dimensional space. Leogao’s original criticism was that the total pool of care for the human oneself is somehow bigger than that of care for ALL same-culture strangers, which is obviously absurd. Similarly, having an AI aligned to you would mean that you are the sole controller of a part of the AI’s values and that the AI can be used to promote your values (e.g. by writing essays expressing them)
As an analogy with inclusive GENETIC fitness. However, genes, unlike memes, are written down in the DNA and usually are transferred to the letter.
The table is likely to be insane, just like Leo Gao is saying. I think this paper by Hendrycks should be considered a rough draft and a very promising direction, rather than something which is “final, polished, and has already solved AI safety”. The table sits in the Appendix where it belongs, Hendrycks should not have brought it up in his headline tweet, it’s a distraction (his theory is not advanced enough to look at this level of its detail, his text is just a starting point, we don’t have to buy all of it).
However, he raises the fundamental question:
what is or who is that “one” whose values these are?
He challenges the assumption that this “one” is all that well-defined.
If we knew which entities are sentient and which are not sentient, in the sense of having subjective reality and such, perhaps we could build a “good enough” theory around that. If we knew that our powerful AIs are going to be definitely sentient, perhaps a world order trying to represent “interests of all sentient beings” would do the job and would be achievable and reliably maintainable. Unfortunately, the “Hard Problem of Consciousness” is unlikely to be solved in time (and also the famous Camp 1 vs Camp 2 disagreements make it difficult to collectively agree that relying on sentience (in the sense of having subjective reality) would be the right approach anyway: https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness).
If we knew that powerful AI systems would be well-defined individuals with long-term persistence, perhaps a world order trying to represent “interests of all individuals” would do the job and would be achievable and reliably maintainable. Unfortunately, AI systems like to fork, merge, and do all kind of morphing, and it’s not clear if we can shape the world mostly in terms of persistent individuals and their interests.
That’s why Hendrycks’ approach is interesting. That approach is trying to achieve something similar without relying on sentience and without relying on stratification into persistent individuals. That makes it quite promising already, at least as a food for thought.
Yes, I agree in substance, but I don’t like how this is phrased. This phrasing is still about dominance (“controller”), about this sharp distinction between “me” and “the other” (cf. “Otherness and control in the age of AGI” sequence by Joe Carlsmith, https://www.lesswrong.com/s/BbAvHtorCZqp97X9W/p/TtkfjskkAurvEN8Fa).
Basically, I think that adversarial approaches will see us wiped out. What does seem promising to me is the class of approaches which try to create overlaps, to eliminate precise boundaries, to make various players to care about “collective interests”. To make relatively stronger players to care about relatively weaker players, because the setup is such that any player might find “themselves” in a relatively weaker position in the future, so they would like to have a world order which would protect them in such an eventuality. In this sense, I really like how Hendrycks’ approach is trying to create strong overlaps from the get go and to make sure that a “part of anyone” is always in a weaker position and can benefit from the world order caring enough about the weak.
So, yes, substantially I agree with
but I would like this to be phrased in terms of collaboration (when it is not phrased in terms of merge), rather than in terms of control.
Of course, his more technical claims are very interesting too. E.g. his claim that
(Appendix A.3 page 22) looks super-interesting, but I have not had the chance to ponder this yet.
I am just starting to internalize the details of what Hendrycks says in various parts of his paper. The general direction seems very promising, but the details would need more time to ponder and to meditate upon.