Unfriendly Natural Intelligence

[Gunnar_Zarncke]

Related to: UFAI, Paperclip maximizer, Reason as memetic immune disorder

A discussion with Stefan (cheers, didn’t get your email, please message me) during the European Community Weekend Berlin fleshed out an idea I had toyed around with for some time:

If a UFAI can wreak havoc by driving simple goals to extremes then so should driving human desires to extremes cause problems. And we should already see this.

Actually we do.

We know that just following our instincts on eating (sugar, fat) is unhealthy. We know that stimulating our pleasure centers more or less directly (drugs) is dangerous. We know that playing certain games can lead to comparable addiction. And the recognition of this has led to a large number of more or less fine-tuned anti-memes e.g. dieting, early drug prevention, helplines. These memes steering us away from such behaviors were selected for because they provided aggregate benefits to the (members of) social (sub) systems they are present in.

Many of these memes have become so self-evident we don’t recognize them as such. Some are essential parts of highly complex social systems. What is the general pattern? Did we catch all the critical cases? Are the existing memes well-suited for the task?How are they related. Many are probably deeply woven into our culture and traditions.

Did we miss any anti-memes?

This last question really is at the core of this post. I think we lack some necessary memes keeping new exploitations of our desires in check. Some new ones result from our society a) having developed the capacity to exploit them and b) the scientific knowledge to know how to do this.

To structure our desires and their exploitation I started with a list which I’d now like to present and ask for discussion and addition to. This list basically takes our complexity of value and looks what happens if one factor is singled out and optimized for too much (where what too much means remains to be discussed).

The list:

• Extreme Status

• Extreme Consumption

  • Extreme Food Consumption

  • Extreme Satisfaction of Desire for Shelter

• Extreme Curiosity

  • Science

  • Games

  • Generic Information

• Extreme Empathy

• Extreme Hate

• Extreme Fear

• Extreme Love/​Romance

• Extreme Sex Drive

• Extreme Awe

• Extreme Pleasure from other Aspects

And before going in detail you might ask yourself how each of these might be driven to extremes—at least compared to what our ancestors would think as described in the Eutopia post.

---

Extreme Status

“power corrupts—absolute power corrupts absolutely”

The craving for power and the reward felt when achieving power must be one of the oldest and has brought about the most complex social structures (we developed from feudal system to democracy after all) and memes to keep it in check. This effectively results in a very high power ladder. The Peter Principle can easily lead to permanent disappointment because the craving is never satisfied despite continuous effort. Nonetheless it appears that there is not much risk here from further exploitation by advancing technology. Or?

Homework question: How can social power systems ensure the feeling of achieving power?

Extreme Consumption

Bringing something into ones possession is rewarding in itself independent of the usefulness of the owned stuff. This can be most easily be seen in the different types of collecting hobbies, messy people and shopping addiction. There is a whole branch of science devoted to the satisfaction of the pleasure from consumption: Neuromarketing. But consumption is well-integrated into our society. Controlling consumption is part of market economy. Fine here it seems.

Extreme Food Consumption

Our evolutionary mostly unchecked desires for sugar and fat and probably a lot of other historically scarce substances has got us—after getting out of starvation—straight into collective obesity. This is well-known. There are lots of anti-memes circulating. E.g. diets. Have a look at Lifestyle interventions for rather helpful memes.

Extreme Satisfaction of Desire for Shelter

This is probably the least of our troubles. Luxury houses probably lead to problems in conjunction with status only.

Extreme Curiosity

I will split this into scientific curiosity, games and other.

Science

Curiosity, despite not feeling like an emotion, is nonetheless a strong emotion once basic desires are satisfied. It is what gave us science. Curiosity drives lots of smart people into the sciences. But curiosity alone has no inherent direction. You may arrive at not very useful topics. And then there is a very high knowledge ladder like in the power/​status case potentially denying you satisfaction during school and university (and later). But at least this has structured itself in a way to be mostly beneficial to society.

Games

This is another thing. I know game addicts. I don’t mean the ones in gambling joints (though this obviously also falls into the category of exploiting desires). I mean WoW and the like. Games are optimized to extract as much pleasure and by extension money out of you as possible. They do so to a large degree by appealing to curiosity, but also status (semi-virtual), ownership (virtual) and probably a lot more. Probably requires a separate post to take this fully apart.

Generic Information

If you are a curios person you can quickly burn lots of time in the knowledge depth of the WWW. This can be in the wake of procrastination but also start out as free time of study. See also this xkcd or this.

Extreme Empathy

This was actually one key point I discussed on the Berlin event. Empathy is an emotion which lets you empathize with beings that are important to you.

“Being-like + important”

That is the pattern your brain locks on (oversimplification; but I hope you basically agree). It doesn’t care what kind of being. That’s the job of your higher faculties to decide. It doesn’t care why that being is important. That’s for other emotions to establish.

It’s mostly other dear human beings. It can be lifestock. It can be animals. Pets. I could tell you a story of a fly that was the important being in question. For some people even things can become being-like.

Hofstadter proposed levels of consciousness (an illustration of the concept is online here. Sorry can’t find a better link) and suggested that you empathize with everything above a certain level. But I disagree. There is no level. It’s just your being+important pattern.

Why do I tell you this? Because empathy drives us to form ethical systems. We form ethical systems that (seem to) match our empathy. And then we apply the ethical system. And if the ethical system derives some (unexpected) result we don’t question it (very strongly) because after all empathy can’t be wrong or can’t it? But it should be checked whether empathy can be led astray as easily as any other emotion.

So if you start from a notion that all humans are important beings (because via positive reinforcement you learned that no human beings in your environment are enemies) then you can empathise will all of those and you arrive at an ethical system that treats all humans as equal (even those far away, even in other universes, just because you never encountered negatives).

Please please note that this is descriptive not normative. I’m unclear myself what this exactly means for my ethical system.

If you additionally learn that all animals are important beings because you are exposed to environmentalism, own a personal pet, never experience the historic normality of butchering animals (but e.g. only perceive pictures of this in the mental frame of violence against animals by persons pictured as brutal) than you might arrive at the point where all (higher) animals are important.

This can be raised as needed to include insects and roots (e.g. Jainism) or even plants.

This of course is in contrast to the other end of the spectrum where only members of the same tribe are ‘important beings’.

Interestingly this kind of empathy satisfaction exists since many centuries—but mostly in the context of religious systems. The resulting food restrictions could be aligned with food availability. The key point here is that today where the overall system of memes may not (yet) be as balanced as the system at that time (though one might wonder what dynamics existed before e.g. the Jainism stabilized).

Extreme Hate

I don’t really have a good model for this, but I guess that terrorism falls into this area. This is kept mostly in check by existing structures obviously. In reason as memetic immune disorder this is actually given as an example.

Extreme Fear

Here examples are also harder to come up with, but all cases of over-protection or loss-aversion probably qualify.

This is especially strong regarding children. See this article for a meme trying to counter that and allow a bit of danger for a lot of other utility. For a lot of our fears there exist a whole market of therapy on the one hand and protection devices and services on the other hand.

The fear of death if let run free leads to exploitation by life insurance (which looks beneficial) on the one end to after-life memes (where I’d include cryonics) on the other end.

Extreme Love/​Romance

Now we slowly come to some obvious points. Your desire for relationships and company should be easily exploitable. Especially nowadays as people don’t any longer run across each other but are mostly distracted by all the other exploitations of their attention. After all attention is a scarce resource. Now that you are robbed of this natural satisfaction it can again be satisfied with specific services by the dating industry in the form of dating sites and speed dating and so on.

As this is already structuring itself anew and as this is no new problem (in earlier times people probably didn’t have much time for dating for other reasons) there should be enough structures to resolve this. Whether this process makes people overall happier is another question.

Extreme Sex Drive

Not much to say here. This desire has been exploited the longest and just because we now have contraceptives, free online porn (which doesn’t seem to have detrimental effects), casual sex sites, legalized prostitution (again) and lots of sex toys this probably doesn’t alter things a lot. Because it is intimately related to reproduction it migh cause surprising selection pressures though (a separate post suggests itself).

Extreme Awe

Awe is an emotion. Thus it should be exploitable albeit I’d guess less so than the other. Nonetheless a few areas quick come to mind:

• Awe for things/​places: Impressive building e.g. churches for proselytizing, marketing and propaganda. Art for selling art (often coupled with curiosity). And tourism.

• Awe for persons: Just look at popular celebrities and super stars and the fandom surrounding them. This is heavily marketed and the fans follow this blindly not knowing that they have been hooked on one of their emotions.

• Awe for both: Apple

Extreme pleasure from other Aspects

These randomly come to mind:

• Music: Did you notice how many people hear music all the time from their devices?

• Physical Exertion: Now that we aren’t any longer regularly exerted from physical work we are prone to have the satisfaction of physical exertion be gamed for in e.g. body building and professional sports.

• Thrill: As we no longer any need for training to cope with threats we instead satisfy this part of curiosity in dangers in amusement parks and possibly extreme sports.

• Collective Identity: I’d think that group-thinking and patriotism has been exploited on the one occasion or other.

Homework: Name two examples occurring in the wake of the internet.

---

This closes the list of exploitable desires for now.

In this way (scientific) reason is a memetic immune disorder. Coming up with new doable/​marketable/​organizable ideas dealing with desired in an environment of quick production and communication cycles leads to inhuman structures. Those will mendel themselves out—but maybe we’d like not to wait for that.

We are already living it a Siren World.

The market economy cannot other than make use of this with e.g. Neuromarketing (see also the Forbes article). And the mean thing is: This may only make you want it but not gain pleasure from it. Fact is: There are markets in everything so also in every desire we have.

Markets in this way are worse than an FAI so I have to wonder whether we actually need an FAI to solve the problems we are currently running into…

To avoid a fatalistic end I suggest that knowing about these memes provides you with a toolkit to build your own counter-memes and open the discussion for this.

[chaosmage]

You’re coupling a list of human desires with a list of things you consider to be problems or wastes of time, and say they’re connected by “too much” of the desire.

You fail to define “too much” (because “what our ancestors would think” is less knowable and more heterogenous than almost any other possible definition), and I think that invalidates your entire post, because it leaves me to presume “too much” is just what you disapprove of.

The idea that there can be “too much” of a desire implies a standard of how much of a desire is right and proper. You won’t find that range out in the physical world, so you’ll have to invent a standard yourself. You could probably invent one that is clearly being violated in all of your examples, if you make it so narrow any strongly emotionally motivated behavior violates it.

However, to be consistent with your quasi-evolutionary hypothesis, human activities that provide aggregate benefits to the (members of) social (sub) systems they are present in should not commonly violate your invented standard of how much of a desire is right and proper. Otherwise, you’re just not distinguishing between desires that cause “Unfriendly Natural Intelligence” and those that don’t, i.e. you’re not contributing a distinction between friendly and unfriendly.

I’m confident you cannot come up with a standard for how much of a desire is good vs. how much of it is bad that meets both of these criteria and isn’t tautological.

Also, your list of drives lacks obvious points like desire to reproduce and desire for mankind to colonize the galaxy, so I’m assuming you left them out because you couldn’t find examples where you felt “too much” of them caused problems.

[Gunnar_Zarncke]

Appealing to “extreme compared to what our ancestors would think” may be unknowable in principle but I’m not the first to suggest this as a guideline.

I agree that this isn’t precise. I’d have assumed that it is possible to infer from context what my intended reading of “too much” or “extreme” might be. But that is the typical mind (or knowledge ) fallacy.

Indeed as I mostly describe extremes in ″behavior″ the problem is that Behavioral Addiction is hard to define even for psychologists (or maybe the phenomenon (or its spread) is too new for them to have arrived at a consensus).

But now that you have challenged me to provide a definition I’m happy to be forced to think a bit deeper about this. I came up with this:

##Quantitative Definitions of Behavior Attachment

One could measure the economic or emotional cost to the actor of changing his behavior. The economic cost could be measured on the individual level y

“what would you pay if you could change/​get rid of the behavior?”

(this assumes the society to stay constant).

Or the economic cost could be measured on the society level:

“what economic costs for the society result from this behavior?”

(this assumes the (treatment of the) behavior to stand in isolation of other behaviors/​structures/​memes).

To derive the latter one would probably have to measure occurrences of actual problematic cases e.g.
the number of incidences over time of impulse control disorder associated with the behavior in question.

If you wanted to link incidences with causes you could compare this with the development of the corresponding industries (per country).

The above prompts the question of how to classify the behaviors but I assume that should be comparatively easy.

Instead of measuring the economic effect for the individual one could try to measure emotions either overall or associated with the behavior.

I don’t think it is healthy if (the same or multiple different) emotions are triggered continuously, erratically or suppressed and outside of your control.
I do not know if there is any chance whether an imbalance induced by society leads to Mood disorder or Personality disorder

• but I woudn’t be surprised. I assume that this both could be measured and possibly linked to behaviors/​structures/​memes.

##Qualitative Classification of Behavior Attachment

The following mostly applies to behaviors that are neither beneficial nor pleasurable overall (by the individual subjective experience), but might also apply to multile disjunct positive behaviors.

I propose the following types of behavior lock-in:

• 1) A person knows that they show a behavior, but cannot get rid of the behavior on their own (e.g. because of its addictedness, or missing knowledge of techniques to do so).

• 2) A person knows that they need help by others, but cannot get rid of the behavior by enlisting the help of other people or institutions (either by not knowing about such an option, no such actors being present or no means to get such persons/​institutions to help).

• 3) A person cannot notice that they show a behavior (with its consequences) and no external person or institution is taking the initiative to help the person to get rid of the behavior (by such actors be missing, actors not knowing about the person or the problem or no means to help).

• 4) The external persons or institutions cannot detect the problem but only its symptoms and act delayed.

• 5) No possibility to address this behavior at all because this problem is inherently inaddressable by human and social systems.

This rests on a sound definition of ‘ability to act’. To avoid issues with ‘free will’ I appeal to an operative definition: An actor has the ability to change his behavior if

• the actor is aware of relevant choices of behavior,

• the actor selects a behavior change,

• the choice is viable from the point of external actors and

• the actor fails reliably at changing his behavior

In case this is applied to populations the last item has to be altered to e.g.

• a significant fraction (in relation to the direness of the problen) of actors facing this choice fail to alter their behavior.

Where direness could be quantified as in the quantitative section above.

##Summary Definition

Given this toolset I’d propose to classify a behavior as “too extreme” if the behavior can be classified as type 3 (no societal means to help) and the behavior has objective dire consequences for the individual (at least long term emotionally scaled by the likelihood of the effects).

Examples:

Currently I think our social means to deal with gaming addiction can be classified as type 4 mostly. Same with internet addiction.

Smoking for comparison was moved by prevention and education from type 4 to type 1 or 2 mostly depending on individual abilities and social environment.

The adoration of fandom is also type 4. This is not actually seens as a bug but as a feature. But if you consider this to have long-term negative effects as I do (at least for some fans), then you might agree that some prevention or symptom classification might be helpful.

I don’t want to rule out all such behaviors—I just want people to have a chance to know what they do. Consequences and awareness.

[Stefan_Schubert]

Interesting post. Reminds me of the Aristotelian slogan “everything in moderation” (I don’t know whether he actually said that, though). A standard problem with that slogan is to define what is “moderate” and what is “extreme”, but here you do provide us with a clear definition: “extreme compared to what our ancestors would think”.

I don’t think that it is always bad to be extreme compared to what our ancestors would think, though. We use systematised, rational thought to an extreme degree compared to our ancestors. Even though this potentially has some negative side-effects (e.g. it could be argued that it leads to a loss of playfulness and spontaneity) it seems that by and large this extreme focus on reason has benefitted humanity enormously.

Do you agree with this? If you do, do have any criterion to distinguish between the traits one should have in moderation, and those one should not?

[Gunnar_Zarncke]

I have started to collect applicable general anti-memes. This is the current list:

• Mindfulness and Mindfulness Meditation

• Slow Parenting

• The motto you already gave: “Everything in moderation, including moderation.” ― Oscar Wilde

• So8res Deregulating Distraction method: “The world is full of distractions, and I have plenty of vices. I am just as susceptible as anyone to binging on TV shows or video games or book series. Instead of trying (and often failing) to stop myself from indulging, I decided to allow myself to indulge whenever I really wanted to.”

[alex_zag_al]

The fear of cults, and the related fear of cults of personality, are antimemes against excessive awe of persons.

[Gunnar_Zarncke]

Yes. Over time anti-memes against (some?) of these effects develop. But nobody guarantees that memetically stable (meme fixation?) societies develop which site in arbitrarily low local minima.

[Metus]

I have a pet hypothesis that almost anything popular or widely consumed/​practiced is some kind of porn, that is an extremely potent stimulant for any positive number of human needs. I do not mean drugs like heroin which is chemically inducing a pleasant state in the brain, but stuff like hamburgers, the recipe of which is designed to be a food porn.

Assuming some truth in this hypothesis, what items can we add to your list? Movies, series and television in general are an example of what I think is story porn. Of course well designed games exploit this need too, but appeal to other emotions too. Thinking about your example of exercise, the converse need to relax is exploited by particularly comfortable home surroundings and furniture.

[SPLH]

An essay from Paul Graham which explores this idea and the future trends:

The Acceleration of Addictiveness

[Gunnar_Zarncke]

A very relevant reference. I see clear differences between addiction and redirection of emotions though. Most of the examples I gave at least don’t look like addiction.

See also this applicable rationality quote: http://​​lesswrong.com/​​lw/​​jsm/​​rationality_quotes_march_2014/​​an1k

[Brillyant]

I don’t like the differences and distinctions people find between “addiction” and “{insert all the things people call addiction besides addiction}”.

As a definition, we could say addiction is the threshold of need whereby an individual will (a) die or (b) be incapacitated without an activity of substance, but even then you’d have trouble defining (b) in my mind.

To Metus’ pet Everything-is-Porn Theory, I’d say at least that many more people are “addicted” to many more things than we realize, it just doesn’t kill, or incapaciate them according to a broad enough definition. I wouldn’t know where to begin with examples—many peoples’ lives are basically controlled by many of the drives in your post.

There people who spend 15+ hours a day on Reddit—addicted? How about 5+ hours on Reddit and another 8 on WoW? Have the 13+ hour/​day Reddit/​WoWers who scrape together enough for rent each month (but have no savings) managed to climb above the threshold for “addiction”?

[ColtInn]

This is my view too. A good portion of the people in my life are addicted to something at any given time by this broader definition. I’ve experienced short periods of it myself ranging from gaming to geeking out way to much on a particular topic at the expense of proper food and sleep. I see it as a result of access to an ever increasing range of pleasure induces experiences at ever lower costs and hyperbolic discounting—too much of a good thing with blinders to future costs.

On what Gunnar_Zarncke has named Extreme Curiosity: “Be a philosopher; but, amidst all your philosophy, be still a man.”. -David Hume

[DanielLC]

Are you using “porn” to mean superstimulus?

That seems kind of an odd terminology, considering that porn isn’t necessarily a superstimulus. They may not have had realistic depictions of nude people in the ancestral environment, but they did have actual nude people.

[MarkL]

Perfectly beautiful nude people, on demand, with infinite rapid novelty is a superstimulus.

[Kenny]

Even the plausible, heavily watered down version of your statement is sufficient:

Attractive nude people, easily accessed, with a large amount of variety, is a super-stimulus.

[Lumifer]

I have a pet hypothesis that almost anything popular or widely consumed/​practiced is some kind of porn, that is an extremely potent stimulant for any positive number of human needs.

That looks like a circular approach. Humans need what they extensively do because they need it.

[Gunnar_Zarncke]

I’m not sure about your use of “porn”.

But yes. Obviously the pleasure derived from watching movies is exploited by the media industry. It is not clear which emotions exactly this exploits. Curiosity for sure. But also what I listed under awe: The pleasure from beauty and of course music. Also the hard to pin down pleasure from humor.

[katydee]

While I like the concept, I think this post needs substantial editing and revision prior to posting on LessWrong.

[Gunnar_Zarncke]

A agree that it is no technical report or well researched article but an opinion/​idea post containing more open than closed ends but this doesn’t appear to a requirement for Main posts by itself. I thought I made into a consistent well-linked whole (more so than is usual in a Discussion post suggesting an idea) so I’m really unclear what kind of editing might be needed. Could you give actionable feedback?

EDIT: Nonetheless I assume you know what you say so I moved it to Discussion.

[katydee]

Could you give actionable feedback?

I would run this post (and future posts) past an editor/​proofreader who is strongly familiar with the English language prior to posting.

[Gunnar_Zarncke]

Thank you. I didn’t expect that. The feedback on this I got earlier wouldn’t have suggested it.

[Kenny]

I agree that it would be improved by editing by, or based on feedback from, someone with stronger English writing skills, I didn’t really mind it that much myself.