I agree that viewing preferences as conditioned on the environment, up to and including the entire history of the observable universe, is a sensible improvement over many more simplistic models that result in clear violations of preference normativity and eliminates many of those violations. My concern is that, given that this is not so obvious as to be the normal way of thinking about preferences in all fields and was nonobvious enough that you had to write a post about the point, this makes me cautious about updating to thinking this is sufficient to make the current value abstraction you use sufficient for purposes of AI alignment. I basically view conditionality of preferences as neutral evidence about the explanatory power of the theory (for the purpose of AI alignment).
Valid point, though conditional meta-preferences are things I’ve already written about, and the issue of being wrong now about what your own preferences would be in the future, is also something I’ve addressed multiple times in different forms. Your example is particularly crisp, though.
I agree that viewing preferences as conditioned on the environment, up to and including the entire history of the observable universe, is a sensible improvement over many more simplistic models that result in clear violations of preference normativity and eliminates many of those violations. My concern is that, given that this is not so obvious as to be the normal way of thinking about preferences in all fields and was nonobvious enough that you had to write a post about the point, this makes me cautious about updating to thinking this is sufficient to make the current value abstraction you use sufficient for purposes of AI alignment. I basically view conditionality of preferences as neutral evidence about the explanatory power of the theory (for the purpose of AI alignment).
Valid point, though conditional meta-preferences are things I’ve already written about, and the issue of being wrong now about what your own preferences would be in the future, is also something I’ve addressed multiple times in different forms. Your example is particularly crisp, though.