Given a policy π we can directly search for an input on which it behaves a certain way.
(I’m sure this point is obvious to Paul, but it wasn’t to me)
We can search for inputs on which a policy behaves badly, which is really helpful for verifying the worst case of a certain policy. But we can’t search for a policy which has a good worst case, because that would require using the black box inside the function passed to the black box, which we can’t do. I think you can also say this as “the black box is an NP oracle, not a Σ2P oracle”.
This still means that we can build a system which in the worst case does nothing, rather than in the worst case is dangerous: we do whatever thing to get some policy, then we search for an input on which it behaves badly, and if one exists we don’t run the policy.
(I’m sure this point is obvious to Paul, but it wasn’t to me)
We can search for inputs on which a policy behaves badly, which is really helpful for verifying the worst case of a certain policy. But we can’t search for a policy which has a good worst case, because that would require using the black box inside the function passed to the black box, which we can’t do. I think you can also say this as “the black box is an NP oracle, not a Σ2P oracle”.
This still means that we can build a system which in the worst case does nothing, rather than in the worst case is dangerous: we do whatever thing to get some policy, then we search for an input on which it behaves badly, and if one exists we don’t run the policy.