That prediction may be true. My argument is that “I know this by introspection” (or, introspection-and-generalization-to-others) is insufficient. For a concrete example, consider your 5-year-old self. I remember some pretty definite beliefs I had about my future self that turned out wrong, and if I ask myself how aligned I am with it I don’t even know how to answer, he just seems way too confused and incoherent.
I think it’s also not absurd that you do have perfect caring in the sense relevant to the argument. This does not require that you don’t make mistakes currently. If you can, with increasing intelligence/information, correct yourself, then the pointer is perfect in the relevant sense. “Caring about the values of person X” is relatively simple and may come out of evolution whereas “those values directly” may not.
That prediction may be true. My argument is that “I know this by introspection” (or, introspection-and-generalization-to-others) is insufficient. For a concrete example, consider your 5-year-old self. I remember some pretty definite beliefs I had about my future self that turned out wrong, and if I ask myself how aligned I am with it I don’t even know how to answer, he just seems way too confused and incoherent.
I think it’s also not absurd that you do have perfect caring in the sense relevant to the argument. This does not require that you don’t make mistakes currently. If you can, with increasing intelligence/information, correct yourself, then the pointer is perfect in the relevant sense. “Caring about the values of person X” is relatively simple and may come out of evolution whereas “those values directly” may not.