Slightly more plausibly, suppose that a set of AI Construction Kits are distributed by lottery, three base-ten digits per ticket. An ACK ends up in the hands of an OB/LW reader, Rational Riana, who constructs the AI to believe that the probability of any lottery ticket winning is 1/1000, and that this probability is independent of the retrospective event of Riana winning.
But Riana believes, and so the AI believes as well, and indeed it happens to be true, that if the lottery had come out differently, the ACK would have ended up in the hands of Superstitious Sally, who believes that lottery tickets in her hand are much more likely than average to win; and Sally’s AI would have believed that the chance of Sally’s next lottery ticket winning was 1⁄10. (Furthermore, Sally’s AI might believe that Sally winning the previous lottery was additional evidence to this effect, but we can leave out that fillip for now.)
It seems to me that it is quite rational for Riana’s AI to believe that the subjunctive Sally’s AI it could have been—if, indeed, one’s reference class is such as to treat this counterfactual entity as an alternative instance of “me”—is merely irrational.
Does this mean that Riana’s AI isn’t pre-rational? Or that Riana’s AI isn’t pre-rational with respect to the lottery ticket? Can Riana’s AI and Sally’s AI agree on the causal circumstances that led to their existence, while still disagreeing on the probability that Sally’s AI’s lottery ticket will win?
I similarly suspect that if I had been born into the Dark Ages, then “I” would have made many far less rational probability assignments; but I think this “alternative” me would have been simply mistaken due to being raised in an even crazier environment, rather than coherently updating a coherent pre-prior based on different data. Am I not pre-rational with respect to my birth date?
Yes, someone who reasonably believes “If I’d have been programmed by a crazy person, I’d have crazy beliefs” is not pre-rational as I defined it. My main purpose is to support my claim that a set of non-crazy people with common belief that they are not crazy do not agree to disagree. People often respond with the claim that non-crazy people can reasonably have different priors—this paper was an attempt to cut off that option.
Does this mean that Riana’s AI isn’t pre-rational?
According to my understanding of Robin’s definition, yes.
Or that Riana’s AI isn’t pre-rational with respect to the lottery ticket?
I don’t think Robin defined what it would mean for someone to be pre-rational “with respect” to something. You’re either pre-rational, or not.
Can Riana’s AI and Sally’s AI agree on the causal circumstances that led to their existence, while still disagreeing on the probability that Sally’s AI’s lottery ticket will win?
I’m not totally sure what you’re asking here. Do you mean can they, assuming they are pre-rational, or just can they in general? I think the answers are no and yes, respectively.
I think the point you’re making is that just saying Riana’s AI and Sally’s AI are both lacking pre-rationality isn’t very satisfactory, and that perhaps we need some way to conclude that Riana’s AI is rational while Sally’s AI is not.
That would be one possible approach to answering the “what to do” question that I asked at the end of my post. Another approach I was thinking about is to apply Nesov’s “trading across possible worlds” idea to this. Riana’s AI could infer that if it were to change its beliefs to be more like Sally’s AI, then due the the symmetry in the situation, Sally’s AI would (counterfactually) change its beliefs to be more like Riana’s AI. This could in some (perhaps most?) circumstances make both of them better off according to their own priors.
I similarly suspect that if I had been born into the Dark Ages, then “I” would have made many far less rational probability assignments;
This example is not directly analogous to the previous one, because the medieval you might agree that the current you is the more rational one, just like the current you might agree that a future you is more rational.
Indeed, worked examples are your friend.
Slightly more plausibly, suppose that a set of AI Construction Kits are distributed by lottery, three base-ten digits per ticket. An ACK ends up in the hands of an OB/LW reader, Rational Riana, who constructs the AI to believe that the probability of any lottery ticket winning is 1/1000, and that this probability is independent of the retrospective event of Riana winning.
But Riana believes, and so the AI believes as well, and indeed it happens to be true, that if the lottery had come out differently, the ACK would have ended up in the hands of Superstitious Sally, who believes that lottery tickets in her hand are much more likely than average to win; and Sally’s AI would have believed that the chance of Sally’s next lottery ticket winning was 1⁄10. (Furthermore, Sally’s AI might believe that Sally winning the previous lottery was additional evidence to this effect, but we can leave out that fillip for now.)
It seems to me that it is quite rational for Riana’s AI to believe that the subjunctive Sally’s AI it could have been—if, indeed, one’s reference class is such as to treat this counterfactual entity as an alternative instance of “me”—is merely irrational.
Does this mean that Riana’s AI isn’t pre-rational? Or that Riana’s AI isn’t pre-rational with respect to the lottery ticket? Can Riana’s AI and Sally’s AI agree on the causal circumstances that led to their existence, while still disagreeing on the probability that Sally’s AI’s lottery ticket will win?
I similarly suspect that if I had been born into the Dark Ages, then “I” would have made many far less rational probability assignments; but I think this “alternative” me would have been simply mistaken due to being raised in an even crazier environment, rather than coherently updating a coherent pre-prior based on different data. Am I not pre-rational with respect to my birth date?
Yes, someone who reasonably believes “If I’d have been programmed by a crazy person, I’d have crazy beliefs” is not pre-rational as I defined it. My main purpose is to support my claim that a set of non-crazy people with common belief that they are not crazy do not agree to disagree. People often respond with the claim that non-crazy people can reasonably have different priors—this paper was an attempt to cut off that option.
I presume this only applies when full disclosure and trust are present?
According to my understanding of Robin’s definition, yes.
I don’t think Robin defined what it would mean for someone to be pre-rational “with respect” to something. You’re either pre-rational, or not.
I’m not totally sure what you’re asking here. Do you mean can they, assuming they are pre-rational, or just can they in general? I think the answers are no and yes, respectively.
I think the point you’re making is that just saying Riana’s AI and Sally’s AI are both lacking pre-rationality isn’t very satisfactory, and that perhaps we need some way to conclude that Riana’s AI is rational while Sally’s AI is not.
That would be one possible approach to answering the “what to do” question that I asked at the end of my post. Another approach I was thinking about is to apply Nesov’s “trading across possible worlds” idea to this. Riana’s AI could infer that if it were to change its beliefs to be more like Sally’s AI, then due the the symmetry in the situation, Sally’s AI would (counterfactually) change its beliefs to be more like Riana’s AI. This could in some (perhaps most?) circumstances make both of them better off according to their own priors.
This example is not directly analogous to the previous one, because the medieval you might agree that the current you is the more rational one, just like the current you might agree that a future you is more rational.