Cool example! But your argument relies on certain vagueness in the definitions of “satisficer” and “maximiser”, that between:
A: an agent “content when it reaches a certain level expected utility”; and
B: “simply a maximiser with a bounded utility function”
(These definitions are from the OP).
Looking at the situation you presented: “A” would recognise the situation as having an expected utility as 9, and be content with it (until she loses the coin toss...). “B” would not distinguish between the utility of 9 and the utility of 10. Neither agent would see a need to self-modify.
Your argument treats Sally as (seeing itself) morphing from “A” before the coin toss to “B” after—this, IMO, invalidates your example.
Cool example! But your argument relies on certain vagueness in the definitions of “satisficer” and “maximiser”, that between:
A: an agent “content when it reaches a certain level expected utility”; and
B: “simply a maximiser with a bounded utility function”
(These definitions are from the OP).
Looking at the situation you presented: “A” would recognise the situation as having an expected utility as 9, and be content with it (until she loses the coin toss...). “B” would not distinguish between the utility of 9 and the utility of 10. Neither agent would see a need to self-modify.
Your argument treats Sally as (seeing itself) morphing from “A” before the coin toss to “B” after—this, IMO, invalidates your example.