I struggled with this for a long time. I forget which of the LW regulars finally explained it simply enough for me. CTD is not “classical decision theory”, as I previously believed, and it does not summarize to “pick highest expected value”. It’s “causal decision theory”, and it optimizes on results based on a (limited) causal model, which does not allow the box contents to be influenced by the (later in time) choice the agent makes.
“naive, expectation-based decision theory” one-boxes based on probability assignments, regardless of causality—it shuts up and multiplies (sum of probability times outcome). But it’s not a formal prediction model (which causality is), so doesn’t help much in designing and exploring artificial agents.
IOW, causal decision theory is only as good as it’s causal model, which is pretty bad for situations like this.
I struggled with this for a long time. I forget which of the LW regulars finally explained it simply enough for me. CTD is not “classical decision theory”, as I previously believed, and it does not summarize to “pick highest expected value”. It’s “causal decision theory”, and it optimizes on results based on a (limited) causal model, which does not allow the box contents to be influenced by the (later in time) choice the agent makes.
“naive, expectation-based decision theory” one-boxes based on probability assignments, regardless of causality—it shuts up and multiplies (sum of probability times outcome). But it’s not a formal prediction model (which causality is), so doesn’t help much in designing and exploring artificial agents.
IOW, causal decision theory is only as good as it’s causal model, which is pretty bad for situations like this.