If we know the inference system to be consistent, this proves that the line of reasoning you describe can’t happen. Indeed this is essentially the way we prove that the diagonal step guarantees that the agent doesn’t infer its decision: if it did, that would make its inference system unsound, and we assume it’s not. So what happens is that if the system proves that it will one-box, it doesn’t prove that two-boxing leads to $10^10, instead it proves something that would make it one-box, such as that two-boxing leads to minus $300.
The system is sound. Therefore, it doesn’t prove (before the agent decides) that the agent will one-box. Presence of the diagonal step guarantees that a proof of the agent one-boxing is not encountered (before the agent decides).
Well, exactly, that’s what I said: the agent is not allowed to prove that it will do a certain action before its decision is made. This is a limitation. My hypothesis: it is not a necessary limitation for a well-behaved consequentialist agent. Here is an attempt at writing an agent without this limitation.
If we know the inference system to be consistent, this proves that the line of reasoning you describe can’t happen. Indeed this is essentially the way we prove that the diagonal step guarantees that the agent doesn’t infer its decision: if it did, that would make its inference system unsound, and we assume it’s not. So what happens is that if the system proves that it will one-box, it doesn’t prove that two-boxing leads to $10^10, instead it proves something that would make it one-box, such as that two-boxing leads to minus $300.
Hmmm. Wait, doesn’t diagonal step immediately make the system inconsistent as soon as the system proves the agent will one-box?
The system is sound. Therefore, it doesn’t prove (before the agent decides) that the agent will one-box. Presence of the diagonal step guarantees that a proof of the agent one-boxing is not encountered (before the agent decides).
Well, exactly, that’s what I said: the agent is not allowed to prove that it will do a certain action before its decision is made. This is a limitation. My hypothesis: it is not a necessary limitation for a well-behaved consequentialist agent. Here is an attempt at writing an agent without this limitation.