I mostly agree with this, though I’m not yet sold on assigning utilities to partial histories using worst-case assumptions. What exactly do you mean by combining partial histories with worst-case assumptions? How is ‘worst case’ defined?
For every partial history, there is a turing machine which turns it into a hellscape in the “undefined future.” This comes with a huge complexity penalty, of course, but these are precisely the sorts of things you need to watch out for when you start maximizing worst case scenarios (which neglect complexity penalties) rather than likely scenarios.
In order to avoid the procrastination paradox, we want our utility function to be upper semicontinuous in the natural topology on histories. Given such a utility function U on the space of infinite histories, there is a natural way to extend it to the space of finite & infinite histories preserving semicontinuity. Namely, we define U(x) = inf U(xy) where x is a finite history and y is an infinite continuation.
However, this prescription is not necessary: we can have an upper semicontinuous function in the space of finite & infinite histories which doesn’t arise in this way. Coming to think about it, it isn’t very attractive since intuitively the universe coming to end is a better outcome than the universe turning into hell.
I mostly agree with this, though I’m not yet sold on assigning utilities to partial histories using worst-case assumptions. What exactly do you mean by combining partial histories with worst-case assumptions? How is ‘worst case’ defined?
For every partial history, there is a turing machine which turns it into a hellscape in the “undefined future.” This comes with a huge complexity penalty, of course, but these are precisely the sorts of things you need to watch out for when you start maximizing worst case scenarios (which neglect complexity penalties) rather than likely scenarios.
In order to avoid the procrastination paradox, we want our utility function to be upper semicontinuous in the natural topology on histories. Given such a utility function U on the space of infinite histories, there is a natural way to extend it to the space of finite & infinite histories preserving semicontinuity. Namely, we define U(x) = inf U(xy) where x is a finite history and y is an infinite continuation.
However, this prescription is not necessary: we can have an upper semicontinuous function in the space of finite & infinite histories which doesn’t arise in this way. Coming to think about it, it isn’t very attractive since intuitively the universe coming to end is a better outcome than the universe turning into hell.