I worry that this equivocates between “we make our values in an ongoing way together” and simpler kinds of aggregation (analogous to voting, linear combinations, etc.). Well, strictly speaking you haven’t spelled out the former position—maybe I should just assert that it’s the only plausible way to interpret the creation of the individual out of “sub-self” entities, and that ultimately I’d like to describe what different individuals do together in this way also.
To explain somewhat further, I’ll assert that entities to which the concept of reinforcement learning is appropriate (say, Q-learners with function approximation, taking some sort of observation as the “state”)--and which I think I am right to construe as exemplifying the sort of sub-self entities that you’d like this discussion to describe—don’t have values. (The truth of that assertion probably depends on how I’m intending some of the words in ways that readers may not find obvious.) Most simply, they don’t really have beliefs; they don’t have anything we could call “world models” in a strong sense. To ask what they’d think of some possible world, we have to invent a way of presenting that world to their observation stream, opening the possibility of things like framing effects. If we find something that looks like incoherence, we don’t have a way of asking them to reflect on it. (We could try to invent ways to do such things, but if we equip them with such, then I want to say that they are no longer “mere” reinforcement learners.)
As such, they can’t vote, assign utilities, Nash bargain, etc. I think that there’s an interesting phenomenon of the arising of selves. I think that economics-flavored discussion of rational agent concepts can often distract from what’s most important, but in cases like this, can point in a productive direction. And here we see the disanalogy between the arising of selves and simple aggregation.
(Indeed, we see the limits of economic-flavored analysis is that fact that what is really missing from reinforcement learners is the space of possible worlds, which is classically taken as the precondition for analysis of rationality rather than something that can be evaluated as rational or irrational.)
I worry that this equivocates between “we make our values in an ongoing way together” and simpler kinds of aggregation (analogous to voting, linear combinations, etc.). Well, strictly speaking you haven’t spelled out the former position—maybe I should just assert that it’s the only plausible way to interpret the creation of the individual out of “sub-self” entities, and that ultimately I’d like to describe what different individuals do together in this way also.
To explain somewhat further, I’ll assert that entities to which the concept of reinforcement learning is appropriate (say, Q-learners with function approximation, taking some sort of observation as the “state”)--and which I think I am right to construe as exemplifying the sort of sub-self entities that you’d like this discussion to describe—don’t have values. (The truth of that assertion probably depends on how I’m intending some of the words in ways that readers may not find obvious.) Most simply, they don’t really have beliefs; they don’t have anything we could call “world models” in a strong sense. To ask what they’d think of some possible world, we have to invent a way of presenting that world to their observation stream, opening the possibility of things like framing effects. If we find something that looks like incoherence, we don’t have a way of asking them to reflect on it. (We could try to invent ways to do such things, but if we equip them with such, then I want to say that they are no longer “mere” reinforcement learners.)
As such, they can’t vote, assign utilities, Nash bargain, etc. I think that there’s an interesting phenomenon of the arising of selves. I think that economics-flavored discussion of rational agent concepts can often distract from what’s most important, but in cases like this, can point in a productive direction. And here we see the disanalogy between the arising of selves and simple aggregation.
(Indeed, we see the limits of economic-flavored analysis is that fact that what is really missing from reinforcement learners is the space of possible worlds, which is classically taken as the precondition for analysis of rationality rather than something that can be evaluated as rational or irrational.)