Thanks so much Edward, I really appreciate these points!
My understanding of a natural kind is a category that reflects how the world is rather than being constructed in service of our particular interests. To me x is a natural kind should mean something like there is a clear boundary that separates things that are x from things that are not x or xness is defined in terms of a clear property that all things possess to a particular extent. I don’t think agency, or being an agent, is really like that. However, I also think that the concept of being a natural kind is probably not, itself, a natural kind, in the sense that what kinds of boundaries or properties we take to be clear is itself something that is determined as much, if not more, by our interests than the fundamental nature of things. So I certainly don’t mean that not being a natural kind is a bad thing. I just think that when we treat agency as being more of a natural kind then it is that this can lead us to the kind of teleological thinking about the ultimate nature of ‘agency’ that I am concerned about here.
My actual inspiration for this argument did not come from ontology though, and I am not sure that talking about it in terms of natural kinds and constructs was the most helpful way of presenting things, it was just the best I could come up with. What I was really thinking about was Derek Parfit’s reductionist account of personal identity. On this view, a person’s identity is not a fundamental fact about the world, there is actually no clear boundary around the self that would work the way we expect it to. Rather, questions about personal identity are really questions about other facts, what Parfit called ‘relation-r’, and my identity over time is really just a useful description of these facts that makes sense in many practical circumstances but can break down in others. I think agency is like this. When we want to explore a systems agency what we really want to do is to understand a bunch of other things about how the system makes decisions and acts on them. Agency is a useful description that summarises these facts in many practical cases (such as human to human interactions), but if we then use that same description and apply it to other kinds of system, like AIs, this description could be leading us astray.
I take your point that my focus on VNM style coherence arguments may be misplaced here and that there are other, more important, arguments I should be considering. I do think that in the general discourse around AI Safety there is an assumption that agents will inevitably be led to VNM utilities but I think you have a lot more direct experience of the discourse there than I do. On the other hand, I do feel like there could be some terminological disagreements here, I agree with you that these are really claims about rationality rather than agency but I think that there is then often a hidden assumption that agents will inevitably seek to become rational. I would love to talk with you more about this!
I think personal identity (as experienced by humans) is largely a rather peculiar fiction, but one that makes a lot of sense as the viewpoint of the mind of an organism subject to evolution (specifically, to relative inclusive evolutionary fitness): the thing it’s actually evolved to track is the genome.
Thanks so much Edward, I really appreciate these points!
My understanding of a natural kind is a category that reflects how the world is rather than being constructed in service of our particular interests. To me x is a natural kind should mean something like there is a clear boundary that separates things that are x from things that are not x or xness is defined in terms of a clear property that all things possess to a particular extent. I don’t think agency, or being an agent, is really like that. However, I also think that the concept of being a natural kind is probably not, itself, a natural kind, in the sense that what kinds of boundaries or properties we take to be clear is itself something that is determined as much, if not more, by our interests than the fundamental nature of things. So I certainly don’t mean that not being a natural kind is a bad thing. I just think that when we treat agency as being more of a natural kind then it is that this can lead us to the kind of teleological thinking about the ultimate nature of ‘agency’ that I am concerned about here.
My actual inspiration for this argument did not come from ontology though, and I am not sure that talking about it in terms of natural kinds and constructs was the most helpful way of presenting things, it was just the best I could come up with. What I was really thinking about was Derek Parfit’s reductionist account of personal identity. On this view, a person’s identity is not a fundamental fact about the world, there is actually no clear boundary around the self that would work the way we expect it to. Rather, questions about personal identity are really questions about other facts, what Parfit called ‘relation-r’, and my identity over time is really just a useful description of these facts that makes sense in many practical circumstances but can break down in others. I think agency is like this. When we want to explore a systems agency what we really want to do is to understand a bunch of other things about how the system makes decisions and acts on them. Agency is a useful description that summarises these facts in many practical cases (such as human to human interactions), but if we then use that same description and apply it to other kinds of system, like AIs, this description could be leading us astray.
I take your point that my focus on VNM style coherence arguments may be misplaced here and that there are other, more important, arguments I should be considering. I do think that in the general discourse around AI Safety there is an assumption that agents will inevitably be led to VNM utilities but I think you have a lot more direct experience of the discourse there than I do. On the other hand, I do feel like there could be some terminological disagreements here, I agree with you that these are really claims about rationality rather than agency but I think that there is then often a hidden assumption that agents will inevitably seek to become rational. I would love to talk with you more about this!
I think personal identity (as experienced by humans) is largely a rather peculiar fiction, but one that makes a lot of sense as the viewpoint of the mind of an organism subject to evolution (specifically, to relative inclusive evolutionary fitness): the thing it’s actually evolved to track is the genome.