One thing that I suspect will become necessary in the adversarial cases (those where two agents’ decisions are dependent on (models of) each other) is some kind of recursion in calculation of outcome. Most of the problematic cases come down to whether the agent under observation or Omega can model the combination of both agents better than the other, including how well A models B modeling A modeling B …
In these cases, the better modeler wins, and at some point, a good DT will recognize that picking the joint win (where the agent gets something and Omega fulfills their contract) is better than an unattainable bigger win (where the agent gets more, but Omega is fooled, but we run out of recursion space before finding the outcome, when we model omega as calculating last (more powerful) in our execution).
I don’t strong-upvote often. This is very cool.
One thing that I suspect will become necessary in the adversarial cases (those where two agents’ decisions are dependent on (models of) each other) is some kind of recursion in calculation of outcome. Most of the problematic cases come down to whether the agent under observation or Omega can model the combination of both agents better than the other, including how well A models B modeling A modeling B …
In these cases, the better modeler wins, and at some point, a good DT will recognize that picking the joint win (where the agent gets something and Omega fulfills their contract) is better than an unattainable bigger win (where the agent gets more, but Omega is fooled, but we run out of recursion space before finding the outcome, when we model omega as calculating last (more powerful) in our execution).