I’m not sure, but was this line:
But, from the first species, we learned a fact which this ship can use to shut down the Earth starline
supposed to read “the Huygens starline”?
Patrick_(orthonormal)
Karma: 90
I was going to say that this (although very good) wasn’t quite Weird enough for your purposes; the principal value of the Baby-Eaters seems to be “individual sacrifice on behalf of the group”, which we’re all too familiar with. I can grok their situation well enough to empathize quickly with the Baby-Eaters. I’d have hoped for something even more foreign at first sight.
Then I checked out the story title again.
Eagerly awaiting the next installments!
(E.g. repeating the mantra “Politics is the Mind-Killer” when tempted to characterize the other side as evil)
Uh, I don’t mean that literally, though doing up a whole Litany of Politics might be fun.
Carl:
Those are instrumental reasons, and could be addressed in other ways.
I wouldn’t want to modify/delete hatred for instrumental reasons, but on behalf of the values that seem to clash almost constantly with hatred. Among those are the values I meta-value, including rationality and some wider level of altruism.
I was trying to point out that giving up big chunks of our personality for instrumental benefits can be a real trade-off.
I agree with that heuristic in general. I would be very cautious regarding the means of ending hatred-as-we-know-it in human nature, and I’m open to the possibility that hatred might be integral (in a way I cannot now see) to the rest of what I value. However, given my understanding of human psychology, I find that claim improbable right now.
My first point was that our values are often the victors of cultural/intellectual/moral combat between the drives given us by the blind idiot god; most of human civilization can be described as the attempt to make humans self-modify away from the drives that lost in the cultural clash. Right now, much of this community values (for example) altruism and rationality over hatred where they conflict, and exerts a certain willpower to keep the other drive vanquished at times. (E.g. repeating the mantra “Politics is the Mind-Killer” when tempted to characterize the other side as evil).
So far, we haven’t seen disaster from this weak self-modification against hatred, and we’ve seen a lot of good (from the perspective of the values we privilege). I take this as some evidence that we can hope to push it farther without losing what we care about (or what we want to care about).
Carl:
I don’t think that automatic fear, suspicion and hatred of outsiders is a necessary prerequisite to a special consideration for close friends, family, etc. Also, yes, outgroup hatred makes cooperation on large-scale Prisoner’s Dilemmas even harder than it generally is for humans.
But finally, I want to point out that we are currently wired so that we can’t get as motivated to face a huge problem if there’s no villain to focus fear and hatred on. The “fighting” circuitry can spur us to superhuman efforts and successes, but it doesn’t seem to trigger without an enemy we can characterize as morally evil.
If a disease of some sort threatened the survival of humanity, governments might put up a fight, but they’d never ask (and wouldn’t receive) the level of mobilization and personal sacrifice that they got during World War II— although if they were crafty enough to say that terrorists caused it, they just might. Concern for loved ones isn’t powerful enough without an idea that an evil enemy threatens them.
Wouldn’t you prefer to have that concern for loved ones be a sufficient motivating force?
Roko:
Not so fast. We like some of our evolved values at the expense of others. Ingroup-outgroup dynamics, the way we’re most motivated only when we have someone to fear and hate: this too is an evolved value, and most of the people here would prefer to do away with it if we can.
The interesting part of moral progress is that the values etched into us by evolution don’t really need to be consistent with each other, so as we become more reflective and our environment changes to force new situations upon us, we realize that they conflict with one another. The analysis of which values have been winning and which have been losing (in different times and places) is another fascinating one...
Doug S:
If the broker believes some investment has a positive expectation this year (and is not very likely to crash terribly), he could advise John Smith to invest in it for a year minus a day, take the proceeds and go to Vegas. If he arrives with $550,000 instead of $500,000, there’s a betting strategy more likely to wind up with $1,000,000 than the original plan.
The balance of risk and reward between the investment part and the Vegas part should have an optimal solution; but since anything over $1,000,000 doesn’t factor nearly as much in John’s utility function, I’d expect he’s not going to bother with investment schemes that have small chances of paying off much more than $1,000,000, and he’d rather look for ones that have significant chances of paying off something in between.
Given your actual reasons for wondering about the world economy in 2040 conditioned on there not having been an extinction/Singularity yet, the survivalist option is actually worth a small hedge bet. If you can go (or convince someone else to go) live in a very remote area, with sufficient skills and resources to continue working quietly on building an FAI if there’s a non-existential global catastrophe, that looks like it has a strongly positive expectation (since in those circumstances, the number of competing AI attempts will probably be few if any).
Now considering the Slump scenarios in which civilization stagnates but survives, it looks like there’s not much prospect of winding up with extra capital in that situation, relative to others; but the capital you acquire might go relatively farther.
I have to say that the fact you’re strongly considering these matters is a bit chilling. I’d be relieved if the reason were that you ascribed probability significantly greater than 1% to a Long Slump, but I suspect it’s because you worry humanity will run out of time in many of the other scenarios before FAI work is finished- reducing you to looking at the Black Swan possibilities within which the world might just be saved.
Sexual Weirdtopia: What goes on consensually behind closed doors doesn’t (usually) affect the general welfare negatively, so it’s not a matter of social concern. However, that particular bundle of biases known as “romantic love” has led to so much chaos in the past that it’s become heavily regulated.
People start out life with the love-module suppressed; but many erstwhile romantics feel that in the right circumstances, this particular self-deception can actually better their lives. If a relationship is going well, the couple (or group, perhaps) can propose to fall in love, and ask the higher authorities for a particular love-mod for their minds.
Every so often, each loving relationship must undergo an “audit” in which they have the love-mods removed and decide whether to put them back in. No unrequited love is allowed; if one party ends it, the other must as well...
A rogue paperclipper in a mostly Friendly world can probably only be stopped by racial prejudice—to a rational creature, it’s always easier to feed him your neighbor than it is to fight him.
A couple of problems with this statement, as I see it:
The word “only”. Forget five minutes— think for five seconds about Third Alternatives. At the very least, wouldn’t an emotion for human-favoritism serve the goal better than an emotion for race-favoritism? Then everyone could cooperate more fully, not just each race by itself.
You could be using “racial prejudice” to mean “species prejudice” or something even wider, but that’s not what the question’s about. Your argument gives no reason for maintaining the current brain architecture, which creates these divisions of allegiance within the normal human race.
Rational agents are doomed to fail because they won’t cooperate enough? I stand with Eliezer: rational agents should WIN. If the inevitable result of noncooperation is eventual destruction, genuinely rational agents WILL find ways to cooperate; the Prisoner’s Dilemma doesn’t operate within every conceivable cooperative enterprise.
You might want to footnote, before anyone starts making noise about the ant example, that colony selection is not a case of group selection but a case of individual selection on the queen (and drones), since the rest of the ants don’t reproduce.
Richard, You’re making the exact point Eliezer just did, about how modeling the effects of intelligence doesn’t generally proceed by running a simulation forward. The “ordinarily” he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.
Vladimir,
Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn’t want to self-modify radically in any direction. There do exist many distinct attractors in the space of ‘abstracted idealized dynamics’, as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.
Eliezer’s claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral frameworks lie in the ‘basin’ of a single attractor; thus the incremental self-modifications of cultural history have an ultimate destination which a powerful AI could deduce.
I suspect, however, that the position is more chaotic than this; that there are distinct avenues of moral progress which will lead us to different attractors. In your terms, since our current right is after all not entirely comprehensive and consistent, we could find that both right1 and right2 are both right extrapolations from right, and that right can’t judge unequivocally which one is better.
I agree— and I balk at the concept of “the” Coherent Extrapolated Volition precisely because I suspect there are many distinct attractors for a moral framework like ours. Since our most basic moral impulses come from the blind idiot god, there’s no reason for them to converge under extrapolation; we have areas of agreement today on certain extrapolations, but the convergence seems to be more a matter of cultural communication. It’s not at all inconceivable that other Everett branches of Earth have made very different forms of moral progress from us, no less consistent with reason or consequences or our moral intuitions.
I’d be very interested, of course, to hear Eliezer’s reasons for believing the contrary.
Well, I find that my metamorality meets those criteria, with one exception.
To reiterate once, I think that the foundations of morality as we understand it are certain evolved impulses like the ones we can find in other primates (maternal love, desire to punish a cheater, etc); these are like other emotions, with one key difference: the social component that we expect and rely on others having the same reaction, and accordingly we experience other emotions as more subjective and our moral impulses as more objective.
Note that when I’m afraid of something, and you’re not, this may surprise me but doesn’t anger me; but if I feel moral outrage at something, and you don’t, then I’m liable to get angry with you.
But of course our moralities aren’t just these few basic impulses. Given our capacity for complex thought and for passing down complex cultures, we’ve built up many systems of morality that try to integrate all these impulses. It’s a testament of the power of conscious thought to reshape our very perceptions of the world that we can get away with this— we foment one moral impulse to restrain another when our system tells us so, and we can work up a moral sentiment in extended contexts when our system tells us to do so. (When we fail to correctly extrapolate and apply our moral system, we later think of this as a moral error.)
Of course, some moral systems cohere logically better than others (which is good if we want to think of them as objective), some have better observable consequences, and some require less strenuous effort at reinterpreting experience. Moving from one moral system to another which improves in some of these areas is generally what we call “moral progress”.
This account has no problems with #2 and #3; I don’t see an “impossible question” suggesting itself (though I’m open to suggestions); the only divergence from your desired properties is that it only claims that we can hardly help but believe that some things are right objectively, whether we want them or not. It’s not impossible for an alien species to evolve to conscious thought without any such concept of objective morality, or with one that differs from ours on the most crucial of points (say, our immediate moral pain at seeing something like us suffer); and there’d be nothing in the universe to say which one of us is “right”.
In essence, I think that Subhan is weakly on the right track, but he doesn’t realize that there are some human impulses stronger than anything we’d call “preference”, or that a mix of moral impulse and reasoning and reclassifying of experience is at stake and is that much more complex than the interactions he supposes. Since we as humans have in common both the first-order moral impulses and the perception that these are objective and thus ought to be logically coherent, we aren’t in fact free to construct our moral systems with too many degrees of freedom.
Sorry for the overlong comment. I’m eager to see what tomorrow’s post will bring...
Hmm. These doubts might seem sophomoric to us, since the “idiot god” of evolution couldn’t conspire against our reasoning with the thoroughness of the Dark Lords of the Matrix. But it makes sense to consider these questions in the course of programming an AI, who will have cause to wonder whether its creators might have intentionally circumscribed its reasoning faculties...
Also, the problem with “cogito, ergo sum” is that it tempts us to posit a self distinct from the act of thinking, thus an immaterial soul, when the best interpretation seems to be that there is no “I” apart from the activity of my brain. I agree with Nietzsche here when he calls it a seductive trick of grammar, imagining that a verb implies a subject in this way.
Silas: Some of the more progressive Christian denominations, perhaps? Most of the elite members have become entirely embarrassed of claiming things like the unique divinity of Jesus, but manage to keep this relatively silent (with the partial exception of defectors like ex-Bishop Spong) so as not to offend the more traditional believers in their communion (who of course know about the elites’ unbelief).
The Episcopal Communion, in particular, is going more into schism the more people start to reveal their real theologies.
I fall closer to the morality-as-preference camp, although I’d add two major caveats.
One is that some of these preferences are deeply programmed into the human brain (i.e. “Punish the cheater” can be found in other primates too), as instincts which give us a qualitatively different emotional response than the instincts for direct satisfaction of our desires. The fact that these instincts feel different from (say) hunger or sexual desire goes a long way towards answering your first question for me. A moral impulse feels more like a perception of an external reality than a statement of a personal preference, so we treat it differently in argument.
The second caveat is that because these feel like perceptions, humans of all times and places have put much effort into trying to reconcile these moral impulses into a coherent perception of an objective moral order, denying some impulses where they conflict and manufacturing moral feeling in cases where we “should” feel it for consistency’s sake. The brain is plastic enough that we can in fact do this to a surprising extent. Now, some reconciliations clearly work better than others from an interior standpoint (i.e. they cause less anguish and cognitive dissonance in the moral agent). This partially answers the second question about moral progress— the act of moving from one attempted framework to one that feels more coherent with one’s stronger moral impulses and with one’s reasoning.
And for the last question, the moral impulses are strong instincts, but sometimes others are stronger; and then we feel the conflict as “doing what we shouldn’t”.
That’s where I stand for now. I’m interested to see your interpretation.
What would I do?
When faced with any choice, I’d try and figure out my most promising options, then trace them out into their different probable futures, being sure to include such factors as an action’s psychological effect on the agent. Then I’d evaluate how much I prefer these futures, acknowledging that I privilege my own future (and the futures of people I’m close to) above others (but not unconditionally), and taking care not to be shortsighted. Then I’d try to choose what seems best under those criteria, applied as rationally as I’m capable of.
You know, the sort of thing that we all do anyway, but often without letting our conscious minds realize it, and thus often with some characteristic errors mixed in.
Interesting. Since people are commenting on fiction vs. non-fiction, it’s interesting to note that my formative books were all non-fiction (paleontology, physics, mathematics, philosophy), and that I now find myself much more easily motivated to try understanding the problems of the world than motivated to try fixing them.
Plural of anecdote, etc, etc.