You might want to footnote, before anyone starts making noise about the ant example, that colony selection is not a case of group selection but a case of individual selection on the queen (and drones), since the rest of the ants don’t reproduce.
Patrick_(orthonormal)
Eliezer,
Every time I think you’re about to say something terribly naive, you surprise me. It looks like trying to design an AI morality is a good way to rid oneself of anthropomorphic notions of objective morality, and to try and see where to go from there.
Although I have to say the potshot at Nietzsche misses the mark; his philosophy is not a resignation to meaninglessness, but an investigation of how to go on and live a human or better-than-human life once the moral void has been recognized. I can’t really explicate or defend him in such a short remark, but I’ll say that most of the people who talk about Nietzsche (including, probably, me) read their own thoughts over his own; be cautious for that reason of dismissing him before reading any of his major works.
Well, now I think I understand why you chose to do the QM series on OB. As it stands, the series is a long explication of one of the most subtle anthropocentric biases out there— the bias in favor of a single world with a single past and future, based on our subjective perception of a single continuous conscious experience. It takes a great deal of effort before most of us are even willing to recognize that assumption as potentially problematic.
Oh, and one doesn’t even have to assume the MWI is true to note this; the single-world bias is irrationally strong in us even if it turns out to correspond to reality.
How do the commenters who justify the usual decisions in the face of certainty and uncertainty with respect to gain and loss account for this part of the post?
There are various other games you can also play with certainty effects. For example, if you offer someone a certainty of $400, or an 80% probability of $500 and a 20% probability of $300, they’ll usually take the $400. But if you ask people to imagine themselves $500 richer, and ask if they would prefer a certain loss of $100 or a 20% chance of losing $200, they’ll usually take the chance of losing $200. Same probability distribution over outcomes, different descriptions, different choices.
Assuming that this experiment has actually been validated, there’s hardly a clearer example of obvious bias than a person’s decision on the exact same circumstance being determined by whether it’s described as certain vs. uncertain gain or certain vs. uncertain loss.
And Eliezer, I have to compliment your writing skills: when faced with people positing a utility of certainty, the first thing that came to my mind was the irrational scale invariance such a concept must have if it fulfills the stated role. But if you’d just stated that, people would have argued to Judgment Day on nuances of the idea, trying to salvage it. Instead, you undercut the counterargument with a concrete reductio ad absurdum, replacing $24,000 with 24,000 lives- which you realized would make your interlocutors uncomfortable about making an incorrect decision for the sake of a state of mind. You seem to have applied a vital principle: we generally change our minds not when a good argument is presented to us, but when it makes us uncomfortable by showing how our existing intuitions conflict.
If and when you publish a book, if the writing is of this quality, I’ll recommend it to the heavens.
Eliezer,
I also think that considering the particular topics is helpful here. In the math book, you were pretty confident the statement was wrong once you discovered a clear formal proof, because essentially there’s nothing more to be said.
On the interpretation of quantum mechanics, since you believe we have almost all the relevant data we’ll ever have (save for observed superpositions of larger and larger objects) and the full criteria to decide between these hypotheses given that information, you again think that disagreement is unfounded.
(I suggest you make an exception in your analysis for Scott Aaronson et al, whose view as I understand it is that progress in his research is more important than holding the Best Justified Interpretation at all times, if the different interpretations don’t have consequences for that research; so he uses whatever one seems most helpful at the moment. This is more like asking a different valid question than getting the wrong answer to a question.)
But on the prospects for General AI in the next century, well, there’s all sort of data you don’t yet have that would greatly help, and others might have it; and updating according to Bayes on that data is intractable without significant assumptions. I think that explains your willingness to hear out Daniel Dennett (albeit with some skepticism).
Finally, I think that when it comes to religion you may be implicitly using the same second-order evaluation I’ve come around to. I still ascribe a nonzero chance to my old religion being true—I didn’t find a knockdown logical flaw or something completely impossible in my experience of the world. I just came to the conclusion I didn’t have a specific reason to believe it above others.
However, I’d refuse to give any such religion serious consideration from now on unless it became more than 50% probable to my current self, because taking up a serious religion changes one’s very practice of rationality by making doubt a disvalue. Spending too much thought on a religion can get you stuck there, and it was hard enough leaving the first time around. That’s a second-order phenomenon different from the others: taking the Copenhagen interpretation for a hypothesis doesn’t strongly prevent you from discarding it later.
My best probability of finding the truth lies in the space of nonreligious answers instead of within any particular religion, so I can’t let myself get drawn in. So I do form an object-level bias against religion (akin to your outright dismissal of Aumann), but it’s one I think is justified on a meta-level.
- 30 Jul 2013 9:43 UTC; 15 points) 's comment on Rationality Quotes from people associated with LessWrong by (
Does anyone else suspect that the last full paragraph is meant to give us the assignment for tomorrow morning?
As for my answers, I think that the particulars of this paradigm shift have to enter into it on some level— because as Eliezer pointed out earlier, the Schrödinger’s Cat thought experiment really should have suggested the possibility of superimposed observers to someone, and from there the MWI doesn’t seem too remote.
So I’d have to ascribe the delay in the MWI proposal in great part to the fact that it doesn’t immediately cohere with our subjective experience of consciousness, and that the physicists were culturally separated from other disciplines (including even philosophy and literature) that were proposing less naive interpretations of consciousness.
Actually, you can’t quite escape the problem of the excluded middle by asserting that “This sentence is false” is not well-formed, or meaningful; because Gödel’s sentence G is a perfectly well-formed (albeit horrifically complicated) statement about the properties of natural numbers which is undecidable in exactly the same way as Epimenides’ paradox.
Mathematicians who prefer to use the law of excluded middle (i.e. most of us, including me) have to affirm that (G or ~G) is indeed a theorem, although neither G nor ~G are theorems! (This doesn’t lead to a contradiction within the system, fortunately, because it’s also impossible to formally prove that neither G nor ~G are theorems.)
Carl:
I don’t think that automatic fear, suspicion and hatred of outsiders is a necessary prerequisite to a special consideration for close friends, family, etc. Also, yes, outgroup hatred makes cooperation on large-scale Prisoner’s Dilemmas even harder than it generally is for humans.
But finally, I want to point out that we are currently wired so that we can’t get as motivated to face a huge problem if there’s no villain to focus fear and hatred on. The “fighting” circuitry can spur us to superhuman efforts and successes, but it doesn’t seem to trigger without an enemy we can characterize as morally evil.
If a disease of some sort threatened the survival of humanity, governments might put up a fight, but they’d never ask (and wouldn’t receive) the level of mobilization and personal sacrifice that they got during World War II— although if they were crafty enough to say that terrorists caused it, they just might. Concern for loved ones isn’t powerful enough without an idea that an evil enemy threatens them.
Wouldn’t you prefer to have that concern for loved ones be a sufficient motivating force?
Roko:
Not so fast. We like some of our evolved values at the expense of others. Ingroup-outgroup dynamics, the way we’re most motivated only when we have someone to fear and hate: this too is an evolved value, and most of the people here would prefer to do away with it if we can.
The interesting part of moral progress is that the values etched into us by evolution don’t really need to be consistent with each other, so as we become more reflective and our environment changes to force new situations upon us, we realize that they conflict with one another. The analysis of which values have been winning and which have been losing (in different times and places) is another fascinating one...
Doug S:
If the broker believes some investment has a positive expectation this year (and is not very likely to crash terribly), he could advise John Smith to invest in it for a year minus a day, take the proceeds and go to Vegas. If he arrives with $550,000 instead of $500,000, there’s a betting strategy more likely to wind up with $1,000,000 than the original plan.
The balance of risk and reward between the investment part and the Vegas part should have an optimal solution; but since anything over $1,000,000 doesn’t factor nearly as much in John’s utility function, I’d expect he’s not going to bother with investment schemes that have small chances of paying off much more than $1,000,000, and he’d rather look for ones that have significant chances of paying off something in between.
Bambi,
The ‘you gotta believe me technology’ remark was probably a reference to the AI-Box Experiment.
Phillip,
None of the defenses you mentioned are safe against something that can out-think their designers, any more than current Internet firewalls are really secure against smart and determined hackers.
And blocking protein nanotech is as limited a defense against AGI as prohibiting boxcutters on airplanes is against general terrorist attack. Eliezer promoted it as the first idea he imagined for getting into physical space, not the only avenue.
Richard, You’re making the exact point Eliezer just did, about how modeling the effects of intelligence doesn’t generally proceed by running a simulation forward. The “ordinarily” he speaks of, I assume, refers to the vast majority of physical systems in the Universe, in which there are no complicated optimization processes (especially intelligences) affecting outcomes on the relevant scales.
Silas: Some of the more progressive Christian denominations, perhaps? Most of the elite members have become entirely embarrassed of claiming things like the unique divinity of Jesus, but manage to keep this relatively silent (with the partial exception of defectors like ex-Bishop Spong) so as not to offend the more traditional believers in their communion (who of course know about the elites’ unbelief).
The Episcopal Communion, in particular, is going more into schism the more people start to reveal their real theologies.
I just wanted to say I’ve benefited greatly from this series, and especially from the last few posts. I’d studied some graduate quantum mechanics, but bailed out before Feynman paths, decoherence, etc; and from what I’d experienced with it, I was beginning to think an intuitive explanation of (one interpretation of) quantum mechanics was nigh-impossible. Thanks for proving me wrong, Eliezer.
The argument (from elegance/Occam’s Razor) for the many-worlds interpretation seems impressively strong, too. I’ll be interested to read the exchanges when you let the one-world advocates have their say.
This matters emotionally, even though it shouldn’t (or seems like it shouldn’t).
Hypothetical money is not treated as equivalent to possessed money.
My point exactly. It’s perfectly understandable that we’ve evolved a “bird in the hand/two in the bush” heuristic, because it makes for good decisions in many common contexts; but that doesn’t prevent it from leading to bad decisions in other contexts. And we should try to overcome it in situations where the actual outcome is of great value to us.
A utility function can take things other than money into account, you know.
As well it should. But how large should you set the utilities of psychology that make you treat two descriptions of the same set of outcomes differently? Large enough to account for a difference of $100 in expected value? $10,000? 10,000 lives?
At some point, you have to stop relying on that heuristic and do the math if you care about making the right decision.
Interesting. Since people are commenting on fiction vs. non-fiction, it’s interesting to note that my formative books were all non-fiction (paleontology, physics, mathematics, philosophy), and that I now find myself much more easily motivated to try understanding the problems of the world than motivated to try fixing them.
Plural of anecdote, etc, etc.
I’m not sure, but was this line:
But, from the first species, we learned a fact which this ship can use to shut down the Earth starline
supposed to read “the Huygens starline”?
I was going to say that this (although very good) wasn’t quite Weird enough for your purposes; the principal value of the Baby-Eaters seems to be “individual sacrifice on behalf of the group”, which we’re all too familiar with. I can grok their situation well enough to empathize quickly with the Baby-Eaters. I’d have hoped for something even more foreign at first sight.
Then I checked out the story title again.
Eagerly awaiting the next installments!
Hmm. These doubts might seem sophomoric to us, since the “idiot god” of evolution couldn’t conspire against our reasoning with the thoroughness of the Dark Lords of the Matrix. But it makes sense to consider these questions in the course of programming an AI, who will have cause to wonder whether its creators might have intentionally circumscribed its reasoning faculties...
Also, the problem with “cogito, ergo sum” is that it tempts us to posit a self distinct from the act of thinking, thus an immaterial soul, when the best interpretation seems to be that there is no “I” apart from the activity of my brain. I agree with Nietzsche here when he calls it a seductive trick of grammar, imagining that a verb implies a subject in this way.
Sexual Weirdtopia: What goes on consensually behind closed doors doesn’t (usually) affect the general welfare negatively, so it’s not a matter of social concern. However, that particular bundle of biases known as “romantic love” has led to so much chaos in the past that it’s become heavily regulated.
People start out life with the love-module suppressed; but many erstwhile romantics feel that in the right circumstances, this particular self-deception can actually better their lives. If a relationship is going well, the couple (or group, perhaps) can propose to fall in love, and ask the higher authorities for a particular love-mod for their minds.
Every so often, each loving relationship must undergo an “audit” in which they have the love-mods removed and decide whether to put them back in. No unrequited love is allowed; if one party ends it, the other must as well...