Resist the Happy Death Spiral

Eliezer Yudkowsky4 Dec 2007 1:15 UTC
84 points
45 commentsLW linkArchive

Once upon a time, there was a man who was convinced that he possessed a Great Idea. Indeed, as the man thought upon the Great Idea more and more, he realized that it was not just a great idea, but the most wonderful idea ever. The Great Idea would unravel the mysteries of the universe, supersede the authority of the corrupt and error-ridden Establishment, confer nigh-magical powers upon its wielders, feed the hungry, heal the sick, make the whole world a better place, etc., etc., etc.

The man was Francis Bacon, his Great Idea was the scientific method, and he was the only crackpot in all history to claim that level of benefit to humanity and turn out to be completely right.1

That’s the problem with deciding that you’ll never admire anything that much: Some ideas really are that good. Though no one has fulfilled claims more audacious than Bacon’s; at least, not yet.

But then how can we resist the happy death spiral with respect to Science itself? The happy death spiral starts when you believe something is so wonderful that the halo effect leads you to find more and more nice things to say about it, making you see it as even more wonderful, and so on, spiraling up into the abyss. What if Science is in fact so beneficial that we cannot acknowledge its true glory and retain our sanity? Sounds like a nice thing to say, doesn’t it? Oh no it’s starting ruuunnnnn . . .

If you retrieve the standard cached deep wisdom for don’t go overboard on admiring science, you will find thoughts like “Science gave us air conditioning, but it also made the hydrogen bomb” or “Science can tell us about stars and biology, but it can never prove or disprove the dragon in my garage.” But the people who originated such thoughts were not trying to resist a happy death spiral. They weren’t worrying about their own admiration of science spinning out of control. Probably they didn’t like something science had to say about their pet beliefs, and sought ways to undermine its authority.

The standard negative things to say about science aren’t likely to appeal to someone who genuinely feels the exultation of science—that’s not the intended audience. So we’ll have to search for other negative things to say instead.

But if you look selectively for something negative to say about science—even in an attempt to resist a happy death spiral—do you not automatically convict yourself of rationalization? Why would you pay attention to your own thoughts, if you knew you were trying to manipulate yourself?

I am generally skeptical of people who claim that one bias can be used to counteract another. It sounds to me like an automobile mechanic who says that the motor is broken on your right windshield wiper, but instead of fixing it, they’ll just break your left windshield wiper to balance things out. This is the sort of cleverness that leads to shooting yourself in the foot. Whatever the solution, it ought to involve believing true things, rather than believing you believe things that you believe are false.

Can you prevent the happy death spiral by restricting your admiration of Science to a narrow domain? Part of the happy death spiral is seeing the Great Idea everywhere—thinking about how Communism could cure cancer if it were only given a chance. Probably the single most reliable sign of a cult guru is that the guru claims expertise, not in one area, not even in a cluster of related areas, but in everything. The guru knows what cult members should eat, wear, do for a living; who they should have sex with; which art they should look at; which music they should listen to . . .

Unfortunately for this plan, most people fail miserably when they try to describe the neat little box that science has to stay inside. The usual trick, “Hey, science won’t cure cancer,” isn’t going to fly. “Science has nothing to say about a parent’s love for their child”—sorry, that’s simply false. If you try to sever science from e.g. parental love, you aren’t just denying cognitive science and evolutionary psychology. You’re also denying Martine Rothblatt’s founding of United Therapeutics to seek a cure for her daughter’s pulmonary hypertension.2 Science is legitimately related, one way or another, to just about every important facet of human existence.

All right, so what’s an example of a false nice claim you could make about science?

One false claim, in my humble opinion, is that science is so wonderful that scientists shouldn’t even try to take ethical responsibility for their work—it will turn out well in the end regardless. It appears to me that this misunderstands the process whereby science benefits humanity. Scientists are human; they have prosocial concerns just like most other other people, and this is at least part of why science ends up doing more good than evil.

But that point is, evidently, not beyond dispute. So here’s a simpler false nice claim: “A cancer patient can be cured just through the publishing of enough journal papers.” Or: “Sociopaths could become fully normal, if they just committed themselves to never believing anything without replicated experimental evidence with p < 0.05.”

The way to avoid believing such statements isn’t an affective cap, deciding that science is only slightly nice. Nor searching for reasons to believe that publishing journal articles causes cancer. Nor believing that science has nothing to say about cancer one way or the other.

Rather, if you know with enough specificity how science works, then you know that while it may be possible for “science to cure cancer,” a cancer patient writing journal papers isn’t going to experience a miraculous remission. That specific proposed chain of cause and effect is not going to work out.

The happy death spiral is only an emotional problem because of a perceptual problem, the halo effect, that makes us more likely to accept future positive claims once we’ve accepted an initial positive claim. We can’t get rid of this effect just by wishing; it will probably always influence us a little. But we can manage to slow down, stop, consider each additional nice claim as an additional burdensome detail, and focus on the specific points of the claim apart from its positiveness.

What if a specific nice claim “can’t be disproven” but there are arguments “both for and against” it? Actually these are words to be wary of in general, because often this is what people say when they’re rehearsing the evidence or avoiding the real weak points. Given the danger of the happy death spiral, it makes sense to try to avoid being happy about unsettled claims—to avoid making them into a source of yet more positive affect about something you liked already.

The happy death spiral is only a big emotional problem because of the overly positive feedback, the ability for the process to go critical. You may not be able to eliminate the halo effect entirely, but you can apply enough critical reasoning to keep the halos subcritical—make sure that the resonance dies out rather than exploding.

You might even say that the whole problem starts with people not bothering to critically examine every additional burdensome detail—demanding sufficient evidence to compensate for complexity, searching for flaws as well as support, invoking curiosity—once they’ve accepted some core premise. Without the conjunction fallacy, there might still be a halo effect, but there wouldn’t be a happy death spiral.3

Even on the nicest Nice Thingies in the known universe, a perfect rationalist who demanded exactly the necessary evidence for every additional (positive) claim would experience no affective resonance. You can’t do this, but you can stay close enough to rational to keep your happiness from spiraling out of control.4

Stuart Armstrong gives closely related advice:5

Cut up your Great Thingy into smaller independent ideas, and treat them as independent.

For instance a marxist would cut up Marx’s Great Thingy into a theory of value of labour, a theory of the political relations between classes, a theory of wages, a theory on the ultimate political state of mankind. Then each of them should be assessed independently, and the truth or falsity of one should not halo on the others. If we can do that, we should be safe from the spiral, as each theory is too narrow to start a spiral on its own.

This, metaphorically, is like keeping subcritical masses of plutonium from coming together. Three Great Ideas are far less likely to drive you mad than one Great Idea. Armstrong’s advice also helps promote specificity: As soon as someone says, “Publishing enough papers can cure your cancer,” you ask, “Is that a benefit of the experimental method, and if so, at which stage of the experimental process is the cancer cured? Or is it a benefit of science as a social process, and if so, does it rely on individual scientists wanting to cure cancer, or can they be self-interested?” Hopefully this leads you away from the good or bad feeling, and toward noticing the confusion and lack of support.

To summarize, you do avoid a Happy Death Spiral by:

  • Splitting the Great Idea into parts;

  • Treating every additional detail as burdensome;

  • Thinking about the specifics of the causal chain instead of the good or bad feelings;

  • Not rehearsing evidence; and

  • Not adding happiness from claims that “you can’t prove are wrong”;

but not by:

  • Refusing to admire anything too much;

  • Conducting a biased search for negative points until you feel unhappy again; or

  • Forcibly shoving an idea into a safe box.

1Bacon didn’t singlehandedly invent science, of course, but he did contribute, and may have been the first to realize the power.

2Successfully, I might add.

3For more background, see “Burdensome Details,” “How Much Evidence Does it Take?”, and “Occam’s Razor” in the previous volume, Map and Territory.

4The really dangerous cases are the ones where any criticism of any positive claim about the Great Thingy feels bad or is socially unacceptable. Arguments are soldiers; any positive claim is a soldier on our side; stabbing your soldiers in the back is treason. Then the chain reaction goes supercritical. More on this later.

5Source: http://​​lesswrong.com/​​lw/​​lm/​​affective_death_spirals/​​gp5.

Eliezer Yudkowsky4 Dec 2007 1:15 UTC
84 points
45 commentsLW linkArchive