1 and 2: Taboo the word “right”. You haven’t come anywhere close to the territory.
Eliezer can use “right” casually in this discussion, thanks to his 10,000-word reduction of it on OB; I may or may not agree with his reduction, but at least I know what sort of evidence would verify or falsify a claim he makes about “right” and “wrong”.
The rest of us should probably be more cautious with those words when verifiability is in the air.
The most important thing for me, basically, was the morality sequence and in particular The Moral Void. I was worrying heavily about whether any of the morals I valued were justified in a universe that lacked Intrinsic Meaning. The Morality sequence (and Nietzsche, incidentally) helped me internalize that it’s OK after all to value certain things— that it’s not irrational to have a morality— that there’s no Universal Judge condemning me for the crime of parochialism if I value myself, my friends, humanity, beauty, knowledge, etc— and that even my flight from value judgments was the result of a slightly more meta value judgment.
Seems probable to me that many potential readers aren’t currently too worried about the Moral Void, but those who are need a pretty substantial push in this direction.
Eliezer Yudkowsky knows exactly how best to respond to this thread; he’s just left it as homework for us.
Hell, I’m not sure why evolution had any reason for us to know what our own emotions are in the first place, which would certainly explain why we have to learn to interpret them, and people vary widely in their ability to do so.
This point is incisive, has important consequences for rationality, and deserves a post (by somebody).
Higher mathematics? Many-Worlds Interpretation? GEB? Evolutionary psychology? These things don’t have massive direct sensory stimuli, but have all sent chills of awe down my spine at some point.
In that case, I don’t disagree on substance. But as a matter of clarity,
“It is only right to craft the law such that one’s actions screen off the parameters of one’s birth.”
seems too casual for good communication; it’s what a person would say who takes “right” and “wrong” to be simple ontologically basic properties of the universe.
On the other hand, the less misleading locutions I can think of would be pretty unwieldy in everyday conversation. I wonder if there’s better language we can use or invent to talk about morality from this perspective...
Very useful! Could it be added to one’s User page, perhaps?
I live in Berkeley.
I always thought that the majority of exposition in your Newcomb example went towards, not “Rationalists should WIN”, but a weaker claim which seems to be a smaller inferential distance from most would-be rationalists:
Rationalists should not systematically lose; whatever systematically loses is not rationality.
(Of course, one needs the logical caveat that we’re not dealing with a pure irrationalist-rewarder; but such things don’t seem to exist in this universe at the moment.)
It seems that all you’re saying is that we need information in order to make good decisions. I really don’t think that’s a controverted or remotely necessary point here.
In situations where there is not enough information to work from (even given perfect Bayesian updating), of course rationalists can make the wrong decision. But across the spectrum of similar possible decisions (where fugu is replaced with some other dish you’ve never tried), making rational use of what info you have should result in a positive expectation.
In a sense there is an implicit pre-commitment in the structure of the problem; while you have not pre-committed to a choice on this specific problem, you are essentially pre-committed to a decision-making algorithm.
That’s an interesting, and possibly fruitful, way of looking at the problem.
Apply the Least Convenient Possible World principle.
Also, I don’t think Eliezer keeps harping on Newcomb’s problem because he anticipates experiencing precisely that scenario. I see several important points that I don’t think have been clearly made (not that I’m the one to do so):
We can choose whether and when to implement certain decision algorithms, including classical causal decision theory (CCDT). This choice may in fact be trivial, or it may be subtle, but it is a worthy question for a rationalist.
Although, for any fixed set of options, implementing CCDT maximizes your return, there are in fact cases where the options you have depend on the outcome of a model of your decision algorithm. I’m not talking about Omega, I’m talking about human social life. We base a large portion of our interactions with others on our anticipations of how they might respond. (This isn’t often done rationally by anyone’s standards, but it can be.)
It gets confusing (in particular, Hofstadterian) here, but a plausibly better outcome might be reached in the Prisoner’s Dilemma by selfish non-strangers mutually modeling the other’s likely decision process, and recognizing that only C-C and D-D are stable outcomes under mutual modeling.
Of course, I still feel a bit uncomfortable with this line of reasoning.
See Arguing “By Definition”. It’s particularly problematic when the definition of “rational” is precisely what’s in dispute.
I think this instinct may in fact be evolutionarily optimized for conflicts between individuals; in most group conflicts in the ancestral environment, you probably already belong to one of the sides.
But yes, it does seem to generalize too readily to conflicts where you personally wouldn’t sway the balance.
EDIT: How could we test any of the above theories? My theory seems to predict that describing the conflict as “one single entity versus another” (and triggering modes of thought optimized for third parties to single combat) will give a stronger underdog bias than describing a collection of entities on each side (with one collection much larger than the other).
Well, I don’t think I’d fare better by thinking less rationally; and if I really needed to find a way to win, rationality at least shouldn’t hurt me me in the process.
I was hoping to be pithy by neglecting a few implicit assumptions. For one, I mean that (in the absence of direct rewards for different cognitive processes) good rationalists shouldn’t systematically lose when they can see a strategy that systematically wins. Of course there are Kobayashi Maru scenarios where all the rationality in the world can’t win, but that’s not what we’re talking about.
As for the last line, I find That Alien Message pretty persuasive. We have plenty of data that we just lack the capacity to process efficiently, either because of bias (i.e. estimating the real-world effectiveness of political policies always turns into a game of selective reporting and analogy) or because of our limited hardware (i.e. protein folding; we have the sequences and the basic interactions, but we can’t currently see the patterns in the data even well enough to pawn off the real work to our computers at any level beyond brute-force). A Bayesian AI with superior hardware would have more than enough data already to crack these basic problems.
It was akrasia, Dostoyevsky, and the sacrament of confession that turned me into a rationalist. Seriously.
I became very religious as a teenager (for social reasons, as I’d later realize), and drifted more and more traditional and conservative (since I could see that liberal Christianity is generally logically incoherent). This drew me into theology (thus philosophy), so that I’d been exposed in college to all the arguments I needed to reject Christianity; I just refused to apply them, generally taking them one at a time and playing One Argument Against an Army.
What changed in grad school had to do with the internalization of the virtue of honesty. Because I had to confess my sins frequently, I became more and more aware of my rationalizations and self-deception (in areas of discipline and akrasia, not of course rationality). I took to heart what Dostoyevsky wrote in “The Brothers Karamazov”:
Above all, do not lie to yourself. A man who lies to himself and listens to his own lie comes to a point where he does not discern any truth either in himself or anywhere around him, and thus falls into disrespect towards himself and others.
Before long, though, the practice of listening for the signs of self-deception and rationalization had an unexpected consequence: my doubts of the faith, which I’d battled as a sin again and again, were growing worse as I recognized the bad arguments I was letting myself be satisfied with. It finally came to the point of recognizing that I was striving to rid myself of doubt when I thought I was striving to investigate it.
From there, it was a comparatively short leap to atheism and to a more consequentialist and physicalist reevaluation of my interpretation of the world. Nietzsche helped greatly, which is why it saddens me when he’s dismissed for the wrong reasons (as invariably happens with people who’ve only heard of him, or only read short bits). But enough about that.
(I later got hooked on OB because the early posts rang very true to what I’d gone through, though I’d never expressed it as clearly as what I saw there.)
My point is that my birth as a rationalist isn’t identical with my fall from religion; it merely caused it as a side effect. My real rationalist beginning was learning to doubt beliefs that felt like they needed no argument, because I’d realized that feelings of certainty arise for reasons besides entanglement with truth.
I’m not convinced that karma is working so well; what’s rewarded by the system is prolific posting at least as much as high-quality posting. Doing away with the auto-self-upvote (and making it impossible to upvote one’s own comments) seems likely to be an improvement. Or maybe making each comment count for (say) 1⁄4 as much as an upvote.
Ditto on this. I also suggest that only post points should count for “Top Contributors”; time spent in the comments is certainly worthy, but it seems to be much cheaper to get points that way.
That’s a creative attempt to avoid really considering Newcomb’s problem; but as I suggested earlier, the noisy real-world applications are real enough to make this a question worth confronting on its own terms.
Least Convenient Possible World: Omega is type (3), and does not offer the game at all if it calculates that its answers turn out to be contradictions (as in your example above). At any rate, you’re not capable of building or obtaining an accurate Omega’ for your private use.
Aside: If Omega sees probability p that you one-box, it puts the million dollars in with probability p, and in either case writes p on a slip of paper in that box. Omega has been shown to be extremely well-calibrated, and its p only differs substantially from 0 or 1 in the case of the jokers who’ve tried using a random process to outwit it. (I always thought this would be an elegant solution to that problem; and note that the expected value of 1-boxing with probability p should then be 1000000p+1000(1-p).)
Yes, these are extra rules of the game. But if these restrictions make rationality impossible, then it doesn’t seem human beings can be rational by your standards (as we’re already being modeled fairly often in social life)— in which case, we’ll take whatever Art is our best hope instead, and call that rationality.
So what do you do in this situation?
(haven’t looked through comments, so this may have been suggested many times over)
In a college-level rationality course, it would be most appropriate for a portion of the grade to be determined by an artificial economy. That is, set up a currency and a (relatively even) starting distribution, add (probabilistic) opportunities for investment (perhaps linked to other important parts of the course) and, most importantly, make defection possible, anonymous and easy. Make it, as much as possible, like a vast array of one-shot (or known number of iterations) Prisoner’s Dilemmas.
Then allow students to organize into institutions with rules. Well-taught rationalists should be able to construct a very strong economy along these lines; poorly-taught ones will be only rational enough not to cooperate out of an irrational sense of honor. A student’s final grade on that component will be the logarithm of their final wealth, curved as little as possible.
It would take a well-designed setup, of course, to ensure that we’re truly measuring rationality and not (say) merely group cameraderie; but I think it could be worked out in a satisfactory way.
The main upshot of this as regards rationality verification: if two different rationality curricula run the same economy setup, a consistently better growth rate of one class economy is evidence of the second kind that more complete rationality is being taught. The students have a much bigger incentive towards their own grade than towards the reputation of the class, so it should be a pretty decent test.