It is certainly true that one should not superficially try to replicate Aumann’s theorem, but should try to replicate the process of the bayesians, namely, to model the other agent and see how the other agent could disagree. Surely this is how we disagree with creationists and customer service agents. Even if they are far from bayesian, we can extract information from their behavior, until we can model them.
But modeling is also what RH was advocating for the philosophers. Inwagen accepts Lewis as a peer, perhaps a superior. Moreover, he accepts him as rationally integrating Inwagen’s arguments. This is exactly where Aumann’s argument applies. In fact, Inwagen does model himself and Lewis and claims (I’ve only read the quoted excerpt) that their disagreement must be due to incommunicable insights. Although Aumann’s framework about modelling the world seems incompatible with the idea of incommunicable insight, I think it is right to worry about symmetry. Possibly this leads us into difficult anthropic territory. But EY is right that we should not respond by simply changing our opinions, but we should try to describe this incommunicable insight and see how it has infected our beliefs.
Anthropic arguments are difficult, but I do not think they are relevant in any of the examples, except maybe the initial superintelligence. In that situation, I would argue in a way that may be your argument about dreaming: if something has a false belief about having a detailed model of the world, there’s not much it can do. You might as well say it is dreaming. (I’m not talking about accuracy, but precision and moreover persistence.)
And you seem to say that if it is dreaming it doesn’t count. When you claim that my bayesian score goes up if I insist that I’m awake whenever I feel I’m awake, you seem to be asserting that my assertions in my dreams don’t count. This seems to be a claim about persistence of identity. Of course, my actions in dreams seem to have less import than my actions when awake, so I should care less about dream error. But I should not discount it entirely.
It is certainly true that one should not superficially try to replicate Aumann’s theorem, but should try to replicate the process of the bayesians, namely, to model the other agent and see how the other agent could disagree. Surely this is how we disagree with creationists and customer service agents. Even if they are far from bayesian, we can extract information from their behavior, until we can model them.
But modeling is also what RH was advocating for the philosophers. Inwagen accepts Lewis as a peer, perhaps a superior. Moreover, he accepts him as rationally integrating Inwagen’s arguments. This is exactly where Aumann’s argument applies. In fact, Inwagen does model himself and Lewis and claims (I’ve only read the quoted excerpt) that their disagreement must be due to incommunicable insights. Although Aumann’s framework about modelling the world seems incompatible with the idea of incommunicable insight, I think it is right to worry about symmetry. Possibly this leads us into difficult anthropic territory. But EY is right that we should not respond by simply changing our opinions, but we should try to describe this incommunicable insight and see how it has infected our beliefs.
Anthropic arguments are difficult, but I do not think they are relevant in any of the examples, except maybe the initial superintelligence. In that situation, I would argue in a way that may be your argument about dreaming: if something has a false belief about having a detailed model of the world, there’s not much it can do. You might as well say it is dreaming. (I’m not talking about accuracy, but precision and moreover persistence.)
And you seem to say that if it is dreaming it doesn’t count. When you claim that my bayesian score goes up if I insist that I’m awake whenever I feel I’m awake, you seem to be asserting that my assertions in my dreams don’t count. This seems to be a claim about persistence of identity. Of course, my actions in dreams seem to have less import than my actions when awake, so I should care less about dream error. But I should not discount it entirely.