IMO, the obvious problem is that the counterfactual oracle (by your definition) is useless. Null prediction != no information. Predicted (simulated) operators of oracle will know that they are simulated, because the oracle returned the null prediction. It might completely invalidate the prediction. I think people who know about being simulated behave pretty differently.
Fix: Maybe it’s possible to use the counterfactuals not with null prediction, but with specific actions of operators, described to the oracle for this particular case. However, I think this requires some additional oracle’s capabilities, which can make it more dangerous. I don’t know.
UPD: And if we have the counterfactual oracle with my proposed fix, and then we use it like “make the list of possible actions, make some utility metric, ask the oracle about the value of a metric for each action in the list, execute action with best predicted results”… o-ops, system “oracle + operators” is Expected Utility Maximizer now! May be not the most dangerous, but only until the operators figure out how to delegate creation of the list to the oracle.
IMO, the obvious problem is that the counterfactual oracle (by your definition) is useless. Null prediction != no information. Predicted (simulated) operators of oracle will know that they are simulated, because the oracle returned the null prediction. It might completely invalidate the prediction. I think people who know about being simulated behave pretty differently.
Fix: Maybe it’s possible to use the counterfactuals not with null prediction, but with specific actions of operators, described to the oracle for this particular case. However, I think this requires some additional oracle’s capabilities, which can make it more dangerous. I don’t know.
UPD: And if we have the counterfactual oracle with my proposed fix, and then we use it like “make the list of possible actions, make some utility metric, ask the oracle about the value of a metric for each action in the list, execute action with best predicted results”… o-ops, system “oracle + operators” is Expected Utility Maximizer now! May be not the most dangerous, but only until the operators figure out how to delegate creation of the list to the oracle.