When an embedded agent—let’s say Emmy—is faced with a choice from a set of actions (Ai)i∈1..n, she is uncertain about what type of decision maker she is in her epistemic position.
For every possible choice Ai among (Ai)i∈1..n Emmy might select, the story would be “it turned out that Emmy is an Ak-er”[1].
If Emmy is able to say confidently “I’m about to act Ak”, this means that she has already made her decision (or that she had no choice in the first place, depending on the situation[2]). The choice therefore takes place during the uncertainty phase, when Emmy might say “I’m not sure who I am. Maybe a A6-er, maybe A2-er… Or am I a A8-er?”.
So we have an agent, Emmy, without any observation whatsoever about her future decision. My take here this is anthropic situation: each kind of Emmy (A1-er, …, An-er) is indiscernible before the end of the decision process.
Conclusion:Anthropics situation is not a weird LW problem, but the essence of any choice.
Hope: If valid, this take could help to deconfuse choice, so on agency and optimizer.
I think the feeling about choice would be interesting to explore here, my guess is it’s about the recent update on identity. If Emmy has known for a long time that she is a certain kind, in such a situation she doesn’t feel her action as a choice, because she is already decided to act in a certain way.
[Question] Choice := Anthropics uncertainty? And potential implications for agency
Epistemic status: half-baked thought
When an embedded agent—let’s say Emmy—is faced with a choice from a set of actions (Ai)i∈1..n, she is uncertain about what type of decision maker she is in her epistemic position.
For every possible choice Ai among (Ai)i∈1..n Emmy might select, the story would be “it turned out that Emmy is an Ak-er”[1].
If Emmy is able to say confidently “I’m about to act Ak”, this means that she has already made her decision (or that she had no choice in the first place, depending on the situation[2]). The choice therefore takes place during the uncertainty phase, when Emmy might say “I’m not sure who I am. Maybe a A6-er, maybe A2-er… Or am I a A8-er?”.
So we have an agent, Emmy, without any observation whatsoever about her future decision. My take here this is anthropic situation: each kind of Emmy (A1-er, …, An-er) is indiscernible before the end of the decision process.
Conclusion: Anthropics situation is not a weird LW problem, but the essence of any choice.
Hope: If valid, this take could help to deconfuse choice, so on agency and optimizer.
For instance, faced with the Newcomb problem, she has to turned to be 1-boxer or 2-boxer (at least in a deterministic setup).
I think the feeling about choice would be interesting to explore here, my guess is it’s about the recent update on identity. If Emmy has known for a long time that she is a certain kind, in such a situation she doesn’t feel her action as a choice, because she is already decided to act in a certain way.