Consider a specific thought experiment where modeling others is important, for example Prisoner’s Dilemma. Even if your opponent is much smarter than you, smart enough to simulate your every thought, you can bargain with them using your access to the outcome-button. You can threaten them with defecting if they think too fast or too difficult for you to follow (in particular, they won’t be able to unconditionally simulate you if they comply). And so you’ll rob them of this first-mover advantage by having your finger on the trigger. Your decision is still your own.
(This idea comes from the discussion of “unintended simulation” thought experiment variant on SIAI decision theory list.)
Consider a specific thought experiment where modeling others is important, for example Prisoner’s Dilemma. Even if your opponent is much smarter than you, smart enough to simulate your every thought, you can bargain with them using your access to the outcome-button. You can threaten them with defecting if they think too fast or too difficult for you to follow (in particular, they won’t be able to unconditionally simulate you if they comply). And so you’ll rob them of this first-mover advantage by having your finger on the trigger. Your decision is still your own.
(This idea comes from the discussion of “unintended simulation” thought experiment variant on SIAI decision theory list.)