Instead of having counterfactual oracles operate as though they do not and cannot exist, why not have them operate as though the predictions of oracles can never be read? Designing them in this way would also allow us to escape the following worry from your post:
″...consider a single oracle predicting under the counterfactual that it does not exist. When it is approached with an important question, it has strong evidence that people want to ask that question to an oracle, and since it does not exist it predicts that a new counterfactual oracle will be built to be asked the question. This process is repeated recursively, with answers propagating back down the chain until they reach a fixed point, which is then output by the original counterfactual oracle.”
If the oracle is operating under the assumption that it does exist but the predictions of oracles can never be read, its evidence that people want to ask questions to an oracle will not give it reason to predict that a second oracle will be constructed (a second one wouldn’t be any more useful than the first) or that the answers a hypothetical second oracle might produce would have downstream causal effects. This approach might have advantages over the alternative proposed in the post because counterfactual scenarios in which the predictions of oracles are never read are less dissimilar from the actual world than counterfactual scenarios in which oracles do not and cannot exist.
It seems to me pretty obvious: in counterfactual world where humans don’t get the answer from the first Oracle, humans say “what the heck” and build a working Oracle that gives answers.
Edit: the trick is in the difference between “this Oracle doesn’t give answer” and “all Oracles don’t give answer”. The first scenario described in this comment, the second scenario requires something like logical counterfactuals.
The idea would be to consider a scenario in which it is something like a law of nature that the predictions of oracles can never be read, in just the same way that the authors are considering a scenario in which it is something like a law of nature that oracles do not and cannot exist.
Instead of having counterfactual oracles operate as though they do not and cannot exist, why not have them operate as though the predictions of oracles can never be read? Designing them in this way would also allow us to escape the following worry from your post:
″...consider a single oracle predicting under the counterfactual that it does not exist. When it is approached with an important question, it has strong evidence that people want to ask that question to an oracle, and since it does not exist it predicts that a new counterfactual oracle will be built to be asked the question. This process is repeated recursively, with answers propagating back down the chain until they reach a fixed point, which is then output by the original counterfactual oracle.”
If the oracle is operating under the assumption that it does exist but the predictions of oracles can never be read, its evidence that people want to ask questions to an oracle will not give it reason to predict that a second oracle will be constructed (a second one wouldn’t be any more useful than the first) or that the answers a hypothetical second oracle might produce would have downstream causal effects. This approach might have advantages over the alternative proposed in the post because counterfactual scenarios in which the predictions of oracles are never read are less dissimilar from the actual world than counterfactual scenarios in which oracles do not and cannot exist.
It seems to me pretty obvious: in counterfactual world where humans don’t get the answer from the first Oracle, humans say “what the heck” and build a working Oracle that gives answers. Edit: the trick is in the difference between “this Oracle doesn’t give answer” and “all Oracles don’t give answer”. The first scenario described in this comment, the second scenario requires something like logical counterfactuals.
Hello,
The idea would be to consider a scenario in which it is something like a law of nature that the predictions of oracles can never be read, in just the same way that the authors are considering a scenario in which it is something like a law of nature that oracles do not and cannot exist.