I have doubts about your conclusion that there must be two systems, one responsible for personal identity and another for an abstract model. See http://lesswrong.com/lw/cze/reply_to_holden_on_tool_ai/8fe5 where I introduce a framework along the lines of Orseau-Ring where a concept of personal identity can be maintained (for the purpose of defining the utility function) while at the same time allowing for extreme flexibility in modeling the universe. I think this serves a reasonable descriptive explanation of what personal identity is. On the other hand, from a normative point of view I think personal identity should be discarded as a fundamental concept
I have doubts about your conclusion that there must be two systems, one responsible for personal identity and another for an abstract model. See http://lesswrong.com/lw/cze/reply_to_holden_on_tool_ai/8fe5 where I introduce a framework along the lines of Orseau-Ring where a concept of personal identity can be maintained (for the purpose of defining the utility function) while at the same time allowing for extreme flexibility in modeling the universe. I think this serves a reasonable descriptive explanation of what personal identity is. On the other hand, from a normative point of view I think personal identity should be discarded as a fundamental concept