What was wrong with specifying an agent that uses “[decision theory] unless it’s sure it’ll make a decision, then it makes the opposite choice”?
Doesn’t that mean the agent never makes a decision?
Not really. It means that the agent never mathematically proves that it will make a decision before it makes it. The agent makes decisions by comparing expected utilities on different actions, not by predicting what it will do.
What was wrong with specifying an agent that uses “[decision theory] unless it’s sure it’ll make a decision, then it makes the opposite choice”?
Doesn’t that mean the agent never makes a decision?
Not really. It means that the agent never mathematically proves that it will make a decision before it makes it. The agent makes decisions by comparing expected utilities on different actions, not by predicting what it will do.