The proof of Theorem 1 is rather unclear: “high scoring” is ill-defined, and increasing the probability of some favorable outcome doesn’t prove imply that the action is good for u since it can also increase the probability of some unfavorable outcome. Instead, you can easily construct by hand a u s.t. Qu(ha)≠Qu(h∅), using only that a≠∅ (just set u to equal 1 for any history with prefix ha and 0 for any history with prefix h∅).
A more explicit construction is “let u evaluate to 1 iff it sees “high scoring” observation o at time t; clearly, its EU is increased. If u_A is this utility, let u instead evaluate to .99 iff it sees o at time t (and 0 otherwise).”
It’s true you could prove it in the way you mentioned (although the history h wouldn’t be supplied to the inner utility calculation), but it isn’t very suggestive for the instrumental convergence / opportunity cost phenomenon I was trying to point at.
The proof of Theorem 1 is rather unclear: “high scoring” is ill-defined, and increasing the probability of some favorable outcome doesn’t prove imply that the action is good for u since it can also increase the probability of some unfavorable outcome. Instead, you can easily construct by hand a u s.t. Qu(ha)≠Qu(h∅), using only that a≠∅ (just set u to equal 1 for any history with prefix ha and 0 for any history with prefix h∅).
A more explicit construction is “let u evaluate to 1 iff it sees “high scoring” observation o at time t; clearly, its EU is increased. If u_A is this utility, let u instead evaluate to .99 iff it sees o at time t (and 0 otherwise).”
It’s true you could prove it in the way you mentioned (although the history h wouldn’t be supplied to the inner utility calculation), but it isn’t very suggestive for the instrumental convergence / opportunity cost phenomenon I was trying to point at.