Is that “what we would want if we were more the person we wanted to be”, or “what we would want if we were more the person a much smarter version of us would want to be”? (My understanding of CEV leans towards the latter, and I think your problem is an instance of the former.)
I’m not sure the two are different in any meaningful way. There person we want to be today isn’t well defined—it takes a smarter intelligence to unwind (CEV) our motivations enough to figure out what we mean by “the person we wanted to be.”
Also, what we would want if we were more the person we wanted to be.
Is that “what we would want if we were more the person we wanted to be”, or “what we would want if we were more the person a much smarter version of us would want to be”? (My understanding of CEV leans towards the latter, and I think your problem is an instance of the former.)
I’m not sure the two are different in any meaningful way. There person we want to be today isn’t well defined—it takes a smarter intelligence to unwind (CEV) our motivations enough to figure out what we mean by “the person we wanted to be.”