Caveat: the following is just my thoughts, and does not necessarily reflect LW consensus, let alone authorial intent, in any way. Indeed, it’s quite likely that both the community and the author actively disagree with everything I say here.
With respect to probability being in the map rather than the territory, this is a big part of how probability is used on LW, and remembering that this is what Eliezer has in mind is important in making sense of what he’s on about in later posts in the sequence. This is also one thing the whole “Bayesianism vs. Frequentism” thing is all about.
Probability, as discussed in the LW context, is a property of propositions relative to bodies of evidence: if we both know I’m blindly pulling a ball out of a jar full of red and black balls, and I have evidence that suggests it’s a 70⁄30 R/B split and you have evidence that suggests it’s a 60⁄40 R/B split, then we have different probabilities of drawing a red ball—which does not in any way affect the frequency with which we will draw a red ball, supposing we pulled balls out repeatedly.
That is, the label “probability” is being remapped to refer to subjective probability, and the whole “subjectively objective” business is just one of a number of rhetorical techniques being used to reinforce that mapping.
As for what is gained… I suspect the ultimate goal is to lay the groundwork for later discussions of meta-ethics, in which “right” is similarly remapped to refer to subjective rightness.
Caveat: the following is just my thoughts, and does not necessarily reflect LW consensus, let alone authorial intent, in any way. Indeed, it’s quite likely that both the community and the author actively disagree with everything I say here.
With respect to probability being in the map rather than the territory, this is a big part of how probability is used on LW, and remembering that this is what Eliezer has in mind is important in making sense of what he’s on about in later posts in the sequence. This is also one thing the whole “Bayesianism vs. Frequentism” thing is all about.
Probability, as discussed in the LW context, is a property of propositions relative to bodies of evidence: if we both know I’m blindly pulling a ball out of a jar full of red and black balls, and I have evidence that suggests it’s a 70⁄30 R/B split and you have evidence that suggests it’s a 60⁄40 R/B split, then we have different probabilities of drawing a red ball—which does not in any way affect the frequency with which we will draw a red ball, supposing we pulled balls out repeatedly.
That is, the label “probability” is being remapped to refer to subjective probability, and the whole “subjectively objective” business is just one of a number of rhetorical techniques being used to reinforce that mapping.
As for what is gained… I suspect the ultimate goal is to lay the groundwork for later discussions of meta-ethics, in which “right” is similarly remapped to refer to subjective rightness.