It is not obvious that vnm rationality is quite the right thing here. I talked to a game theorist dude at an event a while back and he said of the core assumptions of vnm rationality there’s one not everyone accepts, but I’ve had trouble re-locating what it was he said. In any case, garrabrant said he doesn’t accept vnm rationality for related but iirc slightly-different-maybe-not-sure reasons. But iirc it doesn’t completely invalidate combining everything into a single ranking function.
It might make sense to demand that morality is clearly spelled out in one of the theorems that constrain what utility functions we’d accept, though. That’s a little different from it being truly separate.
I concede that VNM may be too high a bar, and I’ve seen some clever things done by relaxing the independence axiom. I don’t think I’ve seen any that change the basic problem of (acting as if there were) a single value function that the agent is optimizing. To be rational, there can be no separation of moral and non-moral domains.
I’m pretty sure I mostly agree. However:
It is not obvious that vnm rationality is quite the right thing here. I talked to a game theorist dude at an event a while back and he said of the core assumptions of vnm rationality there’s one not everyone accepts, but I’ve had trouble re-locating what it was he said. In any case, garrabrant said he doesn’t accept vnm rationality for related but iirc slightly-different-maybe-not-sure reasons. But iirc it doesn’t completely invalidate combining everything into a single ranking function.
It might make sense to demand that morality is clearly spelled out in one of the theorems that constrain what utility functions we’d accept, though. That’s a little different from it being truly separate.
I concede that VNM may be too high a bar, and I’ve seen some clever things done by relaxing the independence axiom. I don’t think I’ve seen any that change the basic problem of (acting as if there were) a single value function that the agent is optimizing. To be rational, there can be no separation of moral and non-moral domains.