Joshua, the thought had occurred to me, but with all due respect to universities, that’s the same sort of training-in-passing that you get from reading “Surely You’re Joking, Mr. Feynman” as a kid. It’s not systematic, and it’s not grounded in the recent advances in cognitive psychology or probability theory. If we continue with the muscle metaphor, then I would say that—if judged by the sole criterion of improving personal skills of rationality—then studying physics is the equivalent of playing tennis. If you actually want to do physics, of course, that’s a whole separate issue. But if you want to study rationality, you really have to study rationality; just as if you wanted to study physics it wouldn’t do to go off and study psychology instead.
Eliezer Yudkowsky
Robin, I’m not sure why you think the difference between “abstract” (?) and non-abstract beliefs is germane to the proper use of humility. It does seem germane to Dennett’s distinction between professing and believing, but that is not the main topic of the essay.
Either I’m missing something, or all of these comments pertain to the general question of why one wants to be rational, with no specialization for the particular question of how to use humility in the service of rationality (assuming from the start that you want to be rational, on which the essay is obviously premised).
Hal, I changed the lead to say “When two or more human beings have common knowledge that they disagree”, which covers your counterexample.
pdf23ds, the prbolem is how to decide when the person you are conversing with is more or equally rational as yourself. What if you disagree about that? Then you have a disagreement about a new variable, your respective degrees of rationality. Do you both believe yourself to be more meta-rational than the other? And so on. See Hanson and Cowen’s “Are Disagreements Honest?”, http://hanson.gmu.edu/deceive.pdf.
Hal, that’s why I specified human beings. Human beings often find themselves with common knowledge that they disagree about a question of fact. And indeed, genuine Bayesians would not find themselves in such a pickle to begin with, which is why I question that we can clean up the mess by imitating the surface features of Bayesians (mutual agreement) while departing from their causal mechanisms (instituting an explicit internal drive to agreement, which is not present in ideal Bayesians).
The reason my addition fixes the problem is that in your scenario, the disagreement only holds while the two observers do not have common knowledge of their own probability estimates—this can easily happen to Bayesians; all they need to do is observe a piece of evidence they haven’t had the opportunity to communicate. So they disagree at first, but only while they don’t have common knowledge.
Hal, I’m not really the best person to explain the Modesty Argument because I don’t believe in it! You should ask a theory’s advocates, not its detractors, to explain it. You, yourself, have advocated that people should agree to agree—how do you think that people should go about it? If your preferred procedure differs from the Modesty Argument as I’ve presented it, it probably means that I got it wrong.
What I mean by the Modesty Argument is: You sit down at a table with someone else who disagrees with you, you each present your first-order arguments about the immediate issue—on the object level, as it were—and then you discover that you still seem to have a disagreement. Then at this point (I consider the Modesty Argument to say), you should consider as evidence the second-order, meta-level fact that the other person isn’t persuaded, and you should take that evidence into account by adjusting your estimate in his direction. And he should do likewise. Keep doing that until you agree.
As to how this fits into Aumann’s original theorem—I’m the wrong person to ask about that, because I don’t think it does fit! But in terms of real-world procedure, I think that’s what Modesty advocates are advocating, more or less. When we’re critiquing Inwagen for failing to agree with Lewis, this is more or less the sort of thing we think he ought to do instead—right?
There are times when I’m happy enough to follow Modest procedure, but the Verizon case, and the creationist case, aren’t on my list. I exercise my individual discretion, and judge based on particular cases. I feel free to not regard a creationist’s beliefs as evidence, despite the apparent symmetry of my belief that he’s the fool and his belief that I’m the fool. Thus I don’t concede that the Modesty Argument holds in general, while Robin Hanson seems (in “Are Disagreements Honest?”) to hold that it should be universal.
Okay, so what are Robin and Hal advocating, procedurally speaking? Let’s hear it from them. I defined the Modesty Argument because I had to say what I thought I was arguing against, but, as said, I’m not an advocate and therefore I’m not the first person to ask. Where do you think Inwagen went wrong in disagreeing with Lewis—what choice did he make that he should not have made? What should he have done instead? The procedure I laid out looks to me like the obvious one—it’s the one I’d follow with a perceived equal. It’s in applying the Modest procedure to disputes about rationality or meta-rationality that I’m likely to start wondering if the other guy is in the same reference class. But if I’ve invented a strawman, I’m willing to hear about it—just tell me the non-strawman version.
Hal, you have to bet at scalar odds. You’ve got to use a scalar quantity to weight the force of your subjective anticipations, and their associated utilities. Giving just the probability, just the betting odds, just the degree of subjective anticipation, does throw away information. More than one set of possible worlds, more than one set of admissible hypotheses, more than one sequence of observable evidence, can yield the final summarized judgment that a certain probability is 1⁄6.
The amount of previously observed evidence can determine how easy it is for additional evidence to shift our beliefs, which in turn determines the expected utility of looking for more information. I think this is what you’re looking for.
But when you have to actually bet, you still bet at 1:5 odds. If that sounds strange to you, look up “ambiguity aversion”—considered a bias—as demonstrated at e.g. http://en.wikipedia.org/wiki/Ellsberg_paradox
PS: Personally I’d bet a lot lower than 1⁄6 on ancient Mars life. And Tom, you’re right that 0 is a safer estimate than 10, but so is 9, and I was assuming the tree was known to be an apple tree in bloom.
Would you say that we’re in basic agreement then that: “Extraordinary claims are always extraordinary evidence, but some claims are more extraordinary evidence than others, and some hypotheses are just too extraordinary.”
Joseph, how did they get these “competing rules” in the first place? By making them up as they went along. So, in accordance with human psychology, they make up lots of different rules for different occasions that “feel different”. Both sides (or all sides) of any religious battle do this, and it doesn’t matter who wins, they still won’t come up with a unified answer.
Tim Worstall, if a PhD economist has pleasurable dreams about winning the lottery, that is exactly what I would call “failing to understand probability on a gut level”. Look at the water! A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million. Otherwise you’ve understood the probability as little symbols on paper but not what it means in real life.
Also, a good economist should be aware that winning the lottery often does not make people happy—though one must take into account that they were the sort of people who bought lottery tickets to begin with.
John, I consider myself a ‘Bayesian wannabe’ and my favorite author thereon is E. T. Jaynes. As such, I follow Jaynes in vehemently denying that the posterior probability following an experiment should depend on “whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved”. See Jaynes’s Probability Theory: The Logic of Science.
The 0.05 significance level is not just “arbitrary”, it is demonstrably too high—in some fields the actual majority of “statistically significant” results fail to replicate, but the failures to replicate don’t get into the prestigious journals, and are not talked about and remembered.
Sorry, ambiguous wording. 0.05 is too weak, and should be replaced with, say, 0.005. It would be a better scientific investment to do fewer studies with twice as many subjects and have nearly all the reported results be replicable. Unfortunately, this change has to be standardized within a field, because otherwise you’re deliberately handicapping yourself in an arms race. This probably deserves its own post.
In my head, I always translate so-called “statistically significant” results into (an often poorly-computed approximation to) a likelihood ratio of 0.05 over the null hypothesis. I believe that experiments should report likelihood ratios.
I am an infinite set atheist—have you ever actually seen an infinite set?
I am a “subjective/objective” Bayesian. If we are ignorant about a phenomenon, this is a fact about our state of mind, not a fact about the phenomenon. Probabilities are in the mind, not in the environment. Nonetheless I follow a correspondence, rather than a coherentist, theory of truth: we are trying to concentrate as much subjective probability mass as possible into (the mental representation that corresponds to) the real state of affairs. See my “The Simple Truth” and “A Technical Explanation of Technical Explanation”.
pdf, that gets into the issue of ignorance priors which is a whole bagful o’ worms in its own right. I tend to believe that we should choose more fundamental and earlier elements of a causal model. The factory was probably built by someone who had in mind a box of a particular volume, and so that, in the real world, is probably the earliest element of the causal model we should be ignorant about. If the factory poofed into existence as a random collection of machinery that happened to manufacture cubic boxes, it would be appropriate to be ignorant about the side length.
Heh. Fair enough, John, I suppose that someone has to arbitrage the books. I’ll add it to Jane Galt’s observation regarding the genuine usefulness of salad forks.
I agree that 0.005 is equally pulled out of a hat. But I also agree on your earlier observation regarding there being some necessity for standardization here.
Personally, I would prefer to standardize “small”, “medium”, and “large” effect sizes, then report likelihood ratios over the point null hypothesis. A very strong advantage of this approach is that it lets someone do a large study and report a startling likelihood advantage of 1000 for “no effect” over “small effect”, rather than just the boring old phrase “not statistically significant”. This is probably worth its own post, but I may not get around to writing it.
The point is not that scientists should be perfect in all spheres of human endeavor. But neither should anyone who really understands science, deliberately start believing things without evidence. It’s not a moral question, merely a gross and indefensible error of cognition. It’s the equivalent of being trained to say that 2 + 2 = 4 on math tests, but when it comes time to add up a pile of candy bars you decide that 2 + 2 ought to equal 5 because you want 5 candy bars. You may do well on math tests, when you apply the rules that have been trained into you, but you don’t understand numbers. Similarly, if you deliberately believe without evidence, you don’t understand cognition or probability theory. You may understand quarks, or cells, but not science.
Newton may have been a hotshot physicist by the standards of the 17th century, but he wasn’t a hotshot rationalist by the standards of this one. (Laplace, on the other hand, was explicitly a probability theorist as well as a physicist, and he was an outstanding rationalist by the standards of that era.)
Yes, there have been many great scientists who believed in utter crap—though fewer of them and weaker belief, as you move toward modern times.
And there have also been many great jugglers who didn’t understand gravity, differential equations, or how their cerebellar cortex learned realtime motor skills. The vast majority of historical geniuses had no idea how their own brains worked, however brainy they may have been.
You can make an amazing discovery, and go down in the historical list of great scientists, without ever understanding what makes Science work. Though you couldn’t build a scientist, just like you couldn’t build a juggler without knowing all that stuff about gravity and differential equations and error correction in realtime motor skills.
I still wouldn’t trust the one’s opinion about a controversial issue in which they had an emotional stake. I couldn’t rely on them to know the difference between evidence versus a wish to believe. If they can compartmentalize their brains for a spirit world, maybe they compartmentalize their brains for scientific controversies too—who knows? If they gave into temptation once, why not again? I’ll find someone else to ask for their summary of the issues.
John Thacker:
I consider myself a finitist, but not an ultrafinitist; I believe in the existence of numbers expressed using Conway chained arrow notation. I am also willing to reject finitism iff a physical theory is constructed which requires me to believe in infinite quantities. I tentatively believe in real numbers and differential equations because physics requires (though I also hold out hope that e.g. holographic physics or some other discrete view may enable me to go digital again). However, I don’t believe that the real numbers in physics are really made of Dedekind cuts, or any other sort of infinite set. I am willing to relinquish my skepticism if a high-energy supercollider breaks open a real number and we find an infinite number of rational numbers bopping around inside it.
I consider the Axiom of Choice to be a work of literary fiction, like “Lord of the Rings”.
Bayesian probability theory works quite well on finite sets. Real-world problems are finite. Why should I need to accept infinity to use Bayes on real-world problems?
The two-envelopes problem shows the necessity of having a finite prior.
Godel’s Completeness theorem shows that any first-order statement true in all models of a set of first-order axioms is provable from those axioms. Thus, the failure of Peano Arithmetic to prove itself consistent is because there are many “supernatural” models of PA in which PA itself is not consistent; that is, there exist supernatural numbers corresponding to proofs of P&~P. PA shouldn’t prove itself consistent because that assertion does not in fact follow from the axioms of PA. (This view was suggested to me by Steve Omohundro.) Now, I don’t believe in these supernatural numbers, but PA hasn’t been given enough information to rule them out, and so it is behaving properly in refusing to assert its own consistency.
I have no desperate psychological need for absolute certainty or proof, which, even if PA proved itself sound, I couldn’t have in any case, because I would have to believe in PA’s soundness before I trusted its proof of soundness. Or maybe I’m in the grips of a Cartesian demon playing with my mathematical abilities.
Correspondence, not coherence, very easily justifies mathematics. Math can make successful predictions, ergo, it’s probably true. No one has ever seen an infinite set, ergo, they probably don’t exist, and at any rate I have no reason to believe in them.
Robin, I agree that the main difficulty is figuring out how to pay off the bets, but it seems to me that—given such a measure—playing a prediction market around the measure makes the game more complex, and hopefully more of a lesson, and more socially involving and personally intriguing. In other words, it’s the difference between “Guess whether it will rain tomorrow?” and “Bob is smiling evilly; are you willing to bet $50 that his probability estimate of 36.3% is too low?” Or to look at it another way, fewer people would play poker if the whole theme was just “Estimate the probability that you can fill an inside straight.” I think Anissimov has a valid fun-amplifying suggestion here.