I agree that this is a major unsolved problem. I started thinking about this problem more than 20 years ago which eventually led to UDT (in part as an attempt to sidestep it). At one point I thought maybe we can just give up anticipation and switch to using UDT which doesn’t depend on a notion of anticipation, but I currently think that some of our values are likely expressed in terms of anticipation so we probably still have to solve the problem (or a version of it) before we can translate them into a UDT utility function.
I agree that this is a major unsolved problem. I started thinking about this problem more than 20 years ago which eventually led to UDT (in part as an attempt to sidestep it). At one point I thought maybe we can just give up anticipation and switch to using UDT which doesn’t depend on a notion of anticipation, but I currently think that some of our values are likely expressed in terms of anticipation so we probably still have to solve the problem (or a version of it) before we can translate them into a UDT utility function.