If the oracle cares about its own performance in a broader sense, rather than just performance on the current question, then don’t we have a problem anyway? E.g. if you ask it question 1, it will be incentivized to make it get an easier question 2? For example, if you are concerned about coordination amongst different instances of the oracle, this seems like it’s a problem regardless.
Yeah, that’s a good point. In my most recent response to Wei Dai I was trying to develop a loss which would prevent that sort of coordination, but it does seem like if that’s happening then it’s a problem in any counterfactual oracle setup, not just this one. Though it is thus still a problem you’d have to solve if you ever actually wanted to implement a counterfactual oracle.
Yeah, that’s a good point. In my most recent response to Wei Dai I was trying to develop a loss which would prevent that sort of coordination, but it does seem like if that’s happening then it’s a problem in any counterfactual oracle setup, not just this one. Though it is thus still a problem you’d have to solve if you ever actually wanted to implement a counterfactual oracle.