Perhaps it would help to introduce a distinction here. Let’s distinguish internal evidence and external evidence. P1 counts as internal evidence for P2 if it is procedurally rational for me to alter my credence in P2 once I come to accept P1, given my background knowledge. P1 is external evidence for P2 if the truth of P1 genuinely counterfactually depends on the truth of P2. That is, P1 would be false (or less frequently true, if we’re dealing with statistical claims) if P2 were false. A proposition can be internal evidence without being external evidence. In your anonymous letter example, the letter is internal evidence but not external evidence.
Which conception of evidence is the right one to use will probably depend on context. When we are attempting to describe an individual’s epistemic status—the amount of reliable information they possess about the world—then it seems that external evidence is the relevant variety of evidence to consider. And if two observers differ substantially in the external evidence available to them, it seems justifiable to place them in separate reference classes for certain anthropic explanations. Going back to an early example of Eliezer’s:
I’m going to close with the thought experiment that initially convinced me of the falsity of the Modesty Argument. In the beginning it seemed to me reasonable that if feelings of 99% certainty were associated with a 70% frequency of true statements, on average across the global population, then the state of 99% certainty was like a “pointer” to 70% probability. But at one point I thought: “What should an (AI) superintelligence say in the same situation? Should it treat its 99% probability estimates as 70% probability estimates because so many human beings make the same mistake?” In particular, it occurred to me that, on the day the first true superintelligence was born, it would be undeniably true that—across the whole of Earth’s history—the enormously vast majority of entities who had believed themselves superintelligent would be wrong. The majority of the referents of the pointer “I am a superintelligence” would be schizophrenics who believed they were God.
A superintelligence doesn’t just believe the bald statement that it is a superintelligence—it presumably possesses a very detailed, very accurate self-model of its own cognitive systems, tracks in detail its own calibration, and so on. But if you tell this to a mental patient, the mental patient can immediately respond: “Ah, but I too possess a very detailed, very accurate self-model!” The mental patient may even come to sincerely believe this, in the moment of the reply. Does that mean the superintelligence should wonder if it is a mental patient? This is the opposite extreme of Russell Wallace asking if a rock could have been you, since it doesn’t know if it’s you or the rock.
If the superintelligence were engaging in anthropic reasoning, should it put itself in the same reference class as the mental patients in all cases? If we think identical (or similar) internal evidence requires that they be in the same reference class, then I think the answer may be yes. But I think the answer is fairly obviously no, and this is because of the vast difference in the epistemic situations of the superintelligence and the mental patients, a difference attributable to differences in external evidence.
I accept your working definitions for “internal evidence” and “external evidence.”
When we are attempting to describe an individual’s epistemic status—the amount of reliable information they possess about the world—then it seems that external evidence is the relevant variety of evidence to consider
I want to be a little careful about the words “epistemic status” and “reliable information,” because a lot of confusion can be introduced through the use of terms that abstract.
I remember reading once that courtship behavior in robins is triggered by the visual stimulus of a patch of red taller than it is wide. I have no idea if this is actually true, but suppose it is. The idea was that the ancestral robin environment didn’t contain other stimuli like that other than female robins in estrus, so it was a reliable piece of evidence to use at the time. Now, of course, there are lots of visual stimuli in that category, so you get robins initiating courtship displays at red socks on clotheslines and at Coke cans.
So, OK. Given that, and using your terms, and assuming it makes any sense to describe what a robin does here as updating on evidence at all, then a vertical red swatch is always internal evidence of a fertile female, and it was external evidence a million years ago (when it “genuinely” counterfactually depended on the presence of such a female) but it is not now. If we put some robins in an environment from which we eliminate all other red things, it would be external evidence again. (Yes?)
If what I am interested in is whether a given robin is correct about whether it’s in the presence of a fertile female, external evidence is the relevant variety of information to consider.
If what I am interested in is what conclusions the robin will actually reach about whether it’s in the presence of a fertile female, internal evidence is the relevant variety of information to consider.
If that is consistent with your claim about the robin’s epistemic status and about the amount of reliable information the robin possesses about the world, then great, I’m with you so far. (If not, this is perhaps a good place to back up and see where we diverged.)
if two observers differ substantially in the external evidence available to them, it seems justifiable to place them in separate reference classes for certain anthropic explanations.
Sure, when available external evidence is particularly relevant to those anthropic explanations.
If the superintelligence were engaging in anthropic reasoning, should it put itself in the same reference class as the mental patients in all cases?
So A and B both believe they’re superintelligences. As it happens, A is in fact a SI, and B is in fact a mental patient. And the question is, should A consider itself in the same reference class as B. Yes?
...I think the answer is fairly obviously no, and this is because of the vast difference in the epistemic situations of the superintelligence and the mental patients,
Absolutely agreed. I don’t endorse any decision theory that results in A concluding that it’s more likely to be a mental patient than a SI in a typical situation like this, and this is precisely because of the nature of the information available to A in such a situation.
If we think identical (or similar) internal evidence requires that they be in the same reference class, then I think the answer may be yes.
Wait, what?
Why in the world would A and B have similar internal evidence?
I mean, in any normal environment, if A is a superintelligence and B is a mental patient, I would expect A to have loads of information on the basis of which it is procedurally rational for A to conclude that A is in a different reference class than B. Which is internal evidence, on your account. No?
But, OK. If I assume that A and B do have similar internal evidence… huh. Well, that implicitly assumes that A is in a pathologically twisted epistemic environment. I have trouble imagining such an environment, but the world is more complex than I can imagine. So, OK, sure, I can assume such an environment, in a suitably hand-waving sort of way.
And sure, I agree with you: in such an environment, A should consider itself in the same reference class as B. A is mistaken, of course, which is no surprise given that it’s in such an epistemically tainted environment.
Now, I suppose one might say something like “Sure, A is justified in doing so, but A should not do so, because A should not believe falsehoods.” Which would reveal a disconnect relating to the word “should,” in addition to everything else. (When I say that A should believe falsehoods in this situation, I mean I endorse the decision procedure that leads to doing so, not that I endorse the result.)
But we at least ought to agree, given your word usage, that it is procedurally rational for A to conclude that it’s in the same reference class as B in such a tainted environment, even though that isn’t true. Yes?
Perhaps it would help to introduce a distinction here. Let’s distinguish internal evidence and external evidence. P1 counts as internal evidence for P2 if it is procedurally rational for me to alter my credence in P2 once I come to accept P1, given my background knowledge. P1 is external evidence for P2 if the truth of P1 genuinely counterfactually depends on the truth of P2. That is, P1 would be false (or less frequently true, if we’re dealing with statistical claims) if P2 were false. A proposition can be internal evidence without being external evidence. In your anonymous letter example, the letter is internal evidence but not external evidence.
Which conception of evidence is the right one to use will probably depend on context. When we are attempting to describe an individual’s epistemic status—the amount of reliable information they possess about the world—then it seems that external evidence is the relevant variety of evidence to consider. And if two observers differ substantially in the external evidence available to them, it seems justifiable to place them in separate reference classes for certain anthropic explanations. Going back to an early example of Eliezer’s:
If the superintelligence were engaging in anthropic reasoning, should it put itself in the same reference class as the mental patients in all cases? If we think identical (or similar) internal evidence requires that they be in the same reference class, then I think the answer may be yes. But I think the answer is fairly obviously no, and this is because of the vast difference in the epistemic situations of the superintelligence and the mental patients, a difference attributable to differences in external evidence.
I accept your working definitions for “internal evidence” and “external evidence.”
I want to be a little careful about the words “epistemic status” and “reliable information,” because a lot of confusion can be introduced through the use of terms that abstract.
I remember reading once that courtship behavior in robins is triggered by the visual stimulus of a patch of red taller than it is wide. I have no idea if this is actually true, but suppose it is. The idea was that the ancestral robin environment didn’t contain other stimuli like that other than female robins in estrus, so it was a reliable piece of evidence to use at the time. Now, of course, there are lots of visual stimuli in that category, so you get robins initiating courtship displays at red socks on clotheslines and at Coke cans.
So, OK. Given that, and using your terms, and assuming it makes any sense to describe what a robin does here as updating on evidence at all, then a vertical red swatch is always internal evidence of a fertile female, and it was external evidence a million years ago (when it “genuinely” counterfactually depended on the presence of such a female) but it is not now. If we put some robins in an environment from which we eliminate all other red things, it would be external evidence again. (Yes?)
If what I am interested in is whether a given robin is correct about whether it’s in the presence of a fertile female, external evidence is the relevant variety of information to consider.
If what I am interested in is what conclusions the robin will actually reach about whether it’s in the presence of a fertile female, internal evidence is the relevant variety of information to consider.
If that is consistent with your claim about the robin’s epistemic status and about the amount of reliable information the robin possesses about the world, then great, I’m with you so far. (If not, this is perhaps a good place to back up and see where we diverged.)
Sure, when available external evidence is particularly relevant to those anthropic explanations.
So A and B both believe they’re superintelligences. As it happens, A is in fact a SI, and B is in fact a mental patient. And the question is, should A consider itself in the same reference class as B. Yes?
Absolutely agreed. I don’t endorse any decision theory that results in A concluding that it’s more likely to be a mental patient than a SI in a typical situation like this, and this is precisely because of the nature of the information available to A in such a situation.
Wait, what?
Why in the world would A and B have similar internal evidence?
I mean, in any normal environment, if A is a superintelligence and B is a mental patient, I would expect A to have loads of information on the basis of which it is procedurally rational for A to conclude that A is in a different reference class than B. Which is internal evidence, on your account. No?
But, OK. If I assume that A and B do have similar internal evidence… huh. Well, that implicitly assumes that A is in a pathologically twisted epistemic environment. I have trouble imagining such an environment, but the world is more complex than I can imagine. So, OK, sure, I can assume such an environment, in a suitably hand-waving sort of way.
And sure, I agree with you: in such an environment, A should consider itself in the same reference class as B. A is mistaken, of course, which is no surprise given that it’s in such an epistemically tainted environment.
Now, I suppose one might say something like “Sure, A is justified in doing so, but A should not do so, because A should not believe falsehoods.” Which would reveal a disconnect relating to the word “should,” in addition to everything else. (When I say that A should believe falsehoods in this situation, I mean I endorse the decision procedure that leads to doing so, not that I endorse the result.)
But we at least ought to agree, given your word usage, that it is procedurally rational for A to conclude that it’s in the same reference class as B in such a tainted environment, even though that isn’t true. Yes?