Robin: would you say that the quantity of addictions—and addictions that make people genuinely, deeply unhappy—in the world is pretty good evidence that we in fact systematically tend to underestimate our self-control problems?
Paul_Gowder
Steve: Wasn’t that the claim of the sophists? “We’ll teach you how to win arguments so you can prevail in politics.” The problem is that the skills for winning arguments aren’t necessarily the schools for rationality in general. Probably the easiest way to learn the skills for winning arguments is to go to law school and “learn to think like a lawyer.”
Eliezer—depends again on whether we’re aggregating across individuals or within one individual. From a utilitarian perspective (see The Post That Is To Come for a non-utilitarian take), that’s my big objection to the specks thing. Slapping each of 100 people once each is not the same as slapping one person 100 times. The first is a series of slaps. The second is a beating.
Honestly, I’m not sure if I’d have given the same answer to all of those questions w/o having heard of the dust specks dilemma. I feel like that world is a little too weird—the thing that motivates me to think about those questions is the dust specks dilemma. They’re not the sort of things practical reason ordinarily has to worry about, or that we can ordinarily expect to have well-developed intuitions about!
Also (and sorry for the rapid-fire commenting), do you accept that we can have conditional probabilities of one? For example, P(A|A)=1? And, for that matter, P(B|(A-->B, A))=1? If so, I believe I can force you to accept at least probabilities of 1 in sound deductive arguments. And perhaps (I’ll have to think about it some more) in the logical laws that get you to the sound deductive arguments. I’m just trying to get the camel’s nose in the tent here...
I agree with Bobvis: a LOT of this is rational:
When University of North Carolina students learned that a speech opposing coed dorms had been banned, they became more opposed to coed dorms (without even hearing the speech). (Probably in Ashmore et. al. 1971.)
This seems straight Bayes to me. The banning of the speech counts as information about the chance that you’ll agree with it, and for a reasonably low probability of banning speech that isn’t dangerous to the administration (i.e. speech that won’t convince), Everyone’s Favorite Probability Rule kicks in and makes it totally rational to become more opposed to coed dorms—assuming, that is, that you believe your chance of being convicted comes largely from rational sources (a belief that practical agents are at least somewhat committed to having).
When a driver said he had liability insurance, experimental jurors awarded his victim an average of four thousand dollars more than if the driver said he had no insurance. If the judge afterward informed the jurors that information about insurance was inadmissible and must be ignored, jurors awarded an average of thirteen thousand dollars more than if the driver had no insurance. (Broeder 1959.)
This too seems rational, though in this case only mostly, not totally. We can understand jurors as trying to balance the costs and the benefits of the award (not their legal job, but a perfectly sane thing to do). And the diminishing marginal utility of wealth suggests that imposing a large judgment on an insurance company causes less disutility to the person paying (or people, distributing that over the company’s clients) than imposing it on a single person. As for the judge’s informing the jurors that insurance information is inadmissible, well, again, they can interpret that instruction as information about the presence of insurance and update accordingly. (Although that might not be accurate in the context of how judges give instructions, jurors need not know that.) Of course, it seems like they updated too much, since they increased their awards much more when p(insurance) increased but is less than 1, than they did when they learned that p(insurance)=1. So it’s still probably partially irrational. But not an artifact of some kind of magical scarcity effect.
I don’t know the literature around Newcomb’s problem very well, so excuse me if this is stupid. BUT: why not just reason as follows:
If the superintelligence can predict your action, one of the following two things must be the case:
a) the state of affairs whether you pick the box or not is already absolutely determined (i.e. we live in a fatalistic universe, at least with respect to your box-picking)
b) your box picking is not determined, but it has backwards causal force, i.e. something is moving backwards through time.
If a), then practical reason is meaningless anyway: you’ll do what you’ll do, so stop stressing about it.
If b), then you should be a one-boxer for perfectly ordinary rational reasons, namely that it brings it about that you get a million bucks with probability 1.
So there’s no problem!
We can go even stronger than mathematical truths. How about the following statement?
~(P &~P)
I think it’s safe to say that if anything is true, that statement (the flipping law of non-contradiction) is true. And it’s the precondition for any other knowledge (for no other reason than if you deny it, you can prove anything). I mean, there are logics that permit contradictions, but then you’re in a space that’s completely alien to normal reasoning.
So that’s lots stronger than 2+2=4. You can reason without 2+2=4. Maybe not very well, but you can do it.
So Eliezer, do you have a probability of 1 in the law of non-contradiction?
Robin: dare I suggest that one area of relevant expertise is normative philosophy for-@#%(^^$-sake?!
It’s just painful—really, really, painful—to see dozens of comments filled with blinkered nonsense like “the contradiction between intuition and philosophical conclusion” when the alleged “philosophical conclusion” hinges on some ridiculous simplistic Benthamite utilitarianism that nobody outside of certain economics departments and insular technocratic computer-geek blog communities actually accepts! My model for the torture case is swiftly becoming fifty years of reading the comments to this post.
The “obviousness” of the dust mote answer to people like Robin, Eliezer, and many commenters depends on the following three claims:
a) you can unproblematically aggregate pleasure and pain across time, space, and individuality,
b) all types of pleasures and pains are commensurable such that for all i, j, given a quantity of pleasure/pain experience i, you can find a quantity of pleasure/pain experience j that is equal to (or greater or less than) it. (i.e. that pleasures and pains exist on one dimension)
c) it is a moral fact that we ought to select the world with more pleasure and less pain.
But each of those three claims is hotly, hotly contested. And almost nobody who has ever thought about the questions seriously believes all three. I expect there are a few (has anyone posed the three beliefs in that form to Peter Singer?), but, man, if you’re a Bayesian and you update your beliefs about those three claims based on the general opinions of people with expertise in the relevant area, well, you ain’t accepting all three. No way, no how.
Tcpkac: wonderful intuition pump.
Gary: interesting—my sense of the nipple piercing case is that yes, there’s a number of unwilling nipple piercings that does add up to 50 years of torture. It might be a number larger than the earth can support, but it exists. I wonder why my intuition is different there. Is yours?
Oh Eliezer, why’d you have to toss that parenthetical in about priors? The rest of the post is so wonderful. But the priors thing… hell, for my part, the objection isn’t to priors that aren’t imposed by some Authority, it’s priors that are completely pulled out of one’s arse. Demanding something beyond the whim of some metaphorical marble bouncing about in one’s brain before one gets to make a probability statement is hardly the same as demanding capital-A-Authority.
gaaahhh. I stop reading for a few days, and on return, find this...
Eliezer, what do these distinctions even mean? I know philosophers who do scary bayesian things, whose work looks a lot—a lot—like math. I know scientists who make vague verbal arguments. I know scientists who work on the “theory” side whose work is barely informed by experiments at all, I know philosophers who are trying to do experiments. It seems like your real distinction is between a priori and a posteriori, and you’ve just flung “philosophy” into the former and “science” into the latter, basically at random.
(I defy you to find an experimental test for Bayes Rule, incidentally—or to utter some non-question-begging statistical principle by which the results could be evaluated.)
If you guys are going to rig elections, I want in.
If you get past that one, I’ll offer you another.
“There is some entity [even if only a simulation] that is having this thought.” Surely you have a probability of 1 in that. Or you’re going to have to answer to Descartes’s upload, yo.
I too see the dust specks as obvious, but for the simpler reason that I reject utilitarian sorts of comparisons like that. Torture is wicked, period. If one must go further, it seems like the suffering from torture is qualitatively worse than the suffering from any number of dust specks.
One possibility: we can see a connection between morality and certain empirical facts—for example, if we believe that more moral societies will be more stable, we might think that we can see moral progress in the form of changes that are brought about by previous morally related instability. That’s not very clear—but a much clearer and more sophisticated variant on that idea can perhaps be seen in an old paper by Joshua Cohen, “The Arc of the Moral Universe” (google scholar will get it, and definitely read it, because a) it’s brilliant, and b) I’m not representing it very well).
Or we might think that some of our morally relevant behaviors are consistently dependent on empirical facts, in which we might progress in finding out. For example, we might have always thought that beings who are as intelligent as we are and have as complex social and emotional lives as do we deserve to be treated as equals. Suppose we think the above at year 1 and year 500, but at year 500, we discover that some group of entities X (which could include fellow humans, as with the slaves, or other species) is as intelligent, etc., and act accordingly. Then it seems like we’ve made clearly directional moral progress—we’ve learned to more accurately make the empirical judgments about which our unchanged moral judgment depends.
Right, but those questions are responsive to reasons too. Here’s where I embrace the recursion. Either we believe that ultimately the reasons stop—that is, that after a sufficiently ideal process, all of the minds in the relevant mind design space agree on the values, or we don’t. If we do, then the superintelligence should replicate that process. If we don’t, then what basis do we have for asking a superintelligence to answer the question? We might as well flip a coin.
Of course, the content of the ideal process is tricky. I’m hiding the really hard questions in there, like what counts as rationality, what kinds of minds are in the relevant mind design space, etc. Those questions are extra-hard because we can’t appeal to an ideal process to answer them on pain of circularity. (Again, political philosophy has been struggling with a version of this question for a very long time. And I do mean struggling—it’s one of the hardest questions there is.) And the best answer I can give is that there is no completely justifiable stopping point: at some point, we’re going to have to declare “these are our axioms, and we’re going with them,” even though those axioms are not going to be justifiable within the system.
What this all comes down to is that it’s all necessarily dependent on social context. The axioms of rationality and the decisions about what constitute relevant mind-space for any such superintelligence would be determined by the brute facts of what kind of reasoning is socially acceptable in the society that creates such a superintelligence. And that’s the best we can do.
Eliezer—no, I don’t think there is. At least, not if the dust specks are distributed over multiple people. Maybe localized in one person—a dust speck every 10th/sec for a sufficiently long period of time might add up to a toe stub.
Peter: Slavery. Genocide.
(Cf. Moore: “here is a hand.”)
I think I’m going to have to write another of my own posts on this (hadn’t I already?), when I have time. Which might not be for a while—which might be never—we’ll see.
For now, let me ask you this Eliezer: often, we think that our intuitions about cases provide a reliable guide to morality. Without that, there’s a serious question about where our moral principles come from. (I, for one, think that question has its most serious bite right on utilitarian moral principles… at least Kant, say, had an argument about how the nature of moral claims leads to his principles.)
So suppose—hypothetically, and I do mean hypothetically—that our best argument for the claim “one ought to maximize net welfare” comes by induction from our intuitions about individual cases. Could we then legitimately use that principle to defend the opposite of our intuitions about cases like this?
More later, I hope.
Eliezer: the rationality of defection in these finitely repeated games has come under some fire, and there’s a HUGE literature on it. Reading some of the more prominent examples may help you sort out your position on it.
Start here:
Robert Aumann. 1995. “Backward Induction and Common Knowledge of Rationality.” Games and Economic Behavior 8:6-19.
Cristina Bicchieri. 1988. “Strategic Behavior and Counterfactuals.” Synthese 76:135-169.
Cristina Bicchieri. 1989. “Self-Refuting Theories of Strategic Interaction: A Paradox of Common Knowledge.” Erkenntnis 30:69-85.
Ken Binmore. 1987. “Modeling Rational Players I.” Economics and Philosophy 3:9-55.
Jon Elster. 1993. “Some unresolved problems in the theory of rational behaviour.” Acta Sociologica 36: 179-190.
Philip Reny. 1992. “Rationality in Extensive-Form Games.” The Journal of Economic Perspectives 6:103-118.
Phillip Petit and Robert Sugden. 1989. “The Backward Induction Paradox.” The Journal of Philosophy 86:169-182.
Brian Skyrms. 1998. “Subjunctive Conditionals and Revealed Preference.” Philosophy of Science 65:545-574
Robert Stalnaker. 1999. “Knowledge, Belief and Counterfactual Reasoning in Games.” in Cristina Bicchieri, Richard Jeffrey, and Brian Skyrms, eds., The Logic of Strategy. New York: Oxford University Press.