Possibly we have different pictures in mind with oracle AIs. I agree that if you train a neural network to imitate the behavior of something non-myopic, then the neural network is itself unlikely to be myopic.
However, I don’t see how the alternative would be useful. That is, the thing that makes it useful to imitate the behavior of something non-myopic tends to be its non-myopic agency. Creating a myopic imitation of its non-myopic agency seems to be self-contradicting goals.
When I imagine oracle AI, I instead more imagine something like “you give the AI a plan and it tells you how the plan would do”. Or “give the AI the current state of the world and it extrapolates the future”. Rather than an imitation-learned or RLHF agent. Is that not the sort of AI others end up imagining?
Admittedly, I agree with you that a solely myopic oracle is best. I just want to warn you that this will be a lot harder than you think to prevent people suggesting solutions that break your assumptions.
Possibly we have different pictures in mind with oracle AIs. I agree that if you train a neural network to imitate the behavior of something non-myopic, then the neural network is itself unlikely to be myopic.
However, I don’t see how the alternative would be useful. That is, the thing that makes it useful to imitate the behavior of something non-myopic tends to be its non-myopic agency. Creating a myopic imitation of its non-myopic agency seems to be self-contradicting goals.
When I imagine oracle AI, I instead more imagine something like “you give the AI a plan and it tells you how the plan would do”. Or “give the AI the current state of the world and it extrapolates the future”. Rather than an imitation-learned or RLHF agent. Is that not the sort of AI others end up imagining?
Admittedly, I agree with you that a solely myopic oracle is best. I just want to warn you that this will be a lot harder than you think to prevent people suggesting solutions that break your assumptions.