jkaufman, the dimmer-switch metaphor of consciousness is intuitively appealing. But consider some of the most intense experiences that humans can undergo, e.g. orgasm, raw agony, or blind panic. Such intense experiences are characterised by a breakdown of any capacity for abstract rational thought or reflective self-awareness. Neuroscanning evidence, too, suggests that much of our higher brain function effectively shuts down during the experience of panic or orgasm. Contrast this intensity of feeling with the subtle and rarefied phenomenology involved in e.g. language production, solving mathematical equations, introspecting one’s thoughts-episodes, etc—all those cognitive capacities that make mature members of our species distinctively human. For sure, this evidence is suggestive, not conclusive. But the supportive evidence converges with e.g. microelectrode studies using awake human subjects. Such studies suggest the limbic brain structures that generate our most intense experiences are evolutionarily very ancient. Also, the same genes, same neurotransmitter pathways and same responses to noxious stimuli are found in our fellow vertebrates. In view of how humans treat nonhumans, I think we ought to be worried that humans could be catastrophically mistaken about nonhuman animal sentience.
davidpearce
First, many apologies Vaniver if you didn’t find any of the AMA topics of interest. But I did also answer—or at least attempt to answer! - questions on Third World poverty; the reproductive revolution; nootropics; scientifically measuring (un)happiness; cognitive bias; brain augmentation; cryonics; empathy enhancement; polyamory; the (alleged) arrogance of transhumanism; Hugo De Garis; Aubrey de Grey; my contribution to Springer’s forthcoming Singularity volume; “hell worlds” and “mangled worlds” in QM; classical versus negative utilitarianism; and meta-ethical anti-realism. If there is an unjustly neglected topic you’d like to see tackled, then I promise I’ll respond to the best of my ability.
Psychedelics? The drug-naive may dismiss their intellectual significance on a priori grounds—and one may abstain altogether on grounds of prudence and/or legality. But will an understanding of consciousness yield to rational philosophical analysis alone, or only to a combination of theory harnessed to empirical methods of investigation? Maybe the former; but I think a defensible case can be made that rigorous experimentation is vital, despite the methodological pitfalls.
Nonhuman animals? Well, the plight of sentient but cognitively humble creatures might seem of limited interest to a community of rationalists like lesswrong. But let’s assume that we take the problem of [human-] Friendly AI seriously. Isn’t the fate of sentient but comparatively cognitively humble creatures in relation to vastly superior intelligence precisely the issue at stake? Should superintelligence(s) care about the well-being of their intellectual inferiors any more than most humans do? And if so, why?
A very nice post. Perhaps you might also discuss Felipe De Brigard’s “Inverted Experience Machine Argument” http://www.unc.edu/~brigard/Xmach.pdf To what extent does our response to Nozick’s Experience Machine Argument typically reflect status quo bias rather than a desire to connect with ultimate reality?
If we really do want to “stay in touch” with reality, then we can’t wirehead or plug into an “Experience Machine”. But this constraint does not rule out radical superhappiness. By genetically recalibrating the hedonic treadmill, we could in principle enjoy rich, intelligent, complex lives based on information-sensitive gradients of bliss—eventually, perhaps, intelligent bliss orders of magnitude richer than anything physiologically accessible today. Optionally, genetic recalibration of our hedonic set-points could in principle leave much if not all of our existing preference architecture intact—defanging Nozick’s Experience Machine Argument—while immensely enriching our quality of life. Radical hedonic recalibration is also easier than, say, the idealised logical reconciliation of Coherent Extrapolated Volition because hedonic recalibration doesn’t entail choosing between mutually inconsistent values—unless of course one’s values are bound up with the inflicting or undergoing suffering.
IMO one big complication with discussions of “wireheading” is that our understanding of intracranial self-stimulation has changed since Olds and Milner discovered the “pleasure centres”. Taking a mu opioid agonist like heroin is in some ways the opposite of wireheading because heroin induces pure bliss without desire (shades of Buddhist nirvana?), whereas intracranial self-stimulation of the mesolimbic dopamine system involves a frenzy of anticipation rather than pure happiness. So it’s often convenient to think of mu opioid agonists as mediating “liking” and dopamine agonists as mediating “wanting”. We have two ultimate cubic centimetre sized “hedonic hotspots” in the rostral shell of the nucleus accumbens and ventral pallidum http://www.lsa.umich.edu/psych/research%26labs/berridge/publications/Berridge%202003%20Brain%20%26%20Cog%20Pleasures%20of%20brain.pdf where mu opioid agonists play a critical signalling role. But anatomical location is critical. Thus the mu opioid agonist remifentanil actually induces dysphoria http://www.ncbi.nlm.nih.gov/pubmed/18801832
the opposite of what one might naively suppose.
Vanvier, do human infants and toddlers deserve moral consideration primarily on account of their potential to become rational adult humans? Or are they valuable in themselves? Young human children with genetic disorders are given love, care and respect—even if the nature of their illness means they will never live to see their third birthday. We don’t hold their lack of “potential” against them. Likewise, pigs are never going to acquire generative syntax or do calculus. But their lack of cognitive sophistication doesn’t make them any less sentient.
Indeed so. Factory-farmed nonhuman animals are debeaked, tail-docked, castrated (etc) to prevent them from mutilating themselves and each other. Self-mutilitary behaviour in particular suggests an extraordinarily severe level of chronic distress. Compare how desperate human beings must be before we self-mutilate. A meat-eater can (correctly) respond that the behavioural and neuroscientific evidence that factory-farmed animals suffer a lot is merely suggestive, not conclusive. But we’re not trying to defeat philosophical scepticism, just act on the best available evidence. Humans who persuade ourselves that factory-farmed animals are happy are simply kidding ourselves—we’re trying to rationalise the ethically indefensible.
Birds lack a neocortex. But members of at least one species, the European magpie, have convincingly passed the “mirror test” [cf. “Mirror-Induced Behavior in the Magpie (Pica pica): Evidence of Self-Recognition” http://www.plosbiology.org/article/fetchObject.action?representation=PDF&uri=info:doi/10.1371/journal.pbio.0060202] Most ethologists recognise passing the mirror test as evidence of a self-concept. As well as higher primates (chimpanzees, orang utans, bonobos, gorillas) members of other species who have passed the mirror test include elephants, orcas and bottlenose dolphins. Humans generally fail the mirror test below the age of eighteen months.
SaidAchmiz, I wonder if a more revealing question would be to ask if / when in vitro meat products of equivalent taste and price hit the market, will you switch? Lesswrong readers tend not to be technophobes, so I assume the majority(?) of lesswrongers who are not already vegetarian will make the transition. However, you say above that you are “not interested in reducing the suffering of animals”. Do you mean that you are literally indifferent one way or the other to nonhuman animal suffering—in which case presumably you won’t bother changing to the cruelty-free alternative? Or do you mean merely that you don’t consider nonhuman animal suffering important?
Vanvier, you say that you wouldn’t be averse to a quick end for young human children who are not going to live to see their third birthday. What about intellectually handicapped children with potentially normal lifespans whose cognitive capacities will never surpass a typical human toddler or mature pig?
Shminux, we’ve all had the experience of making a point we regard as luminously self-evident—and then feeling baffled when someone doesn’t “get” what is foot-stampingly obvious. Is this guy a knave or a fool?! Anyhow, sorry if you think I’m a “terminal case” with “a total lack of interest in learning from the points others make”. If I don’t always respond, often it’s either because I agree, or because I don’t feel I have anything interesting to add—or in the case of Eliezer’s contribution above beginning “Aargh!” [a moan of pleasure?] because I am still mulling over a reply. The delay doesn’t mean I’m ignoring it. Is there is some particular point you’ve made that you feel I’ve unjustly neglected and you’d like an answer to? If so, I’ll do my fallible best to respond.
Just a note about “mind uploading”. On pain of “strong” emergence, classical Turing machines can’t solve the phenomenal binding problem. Their ignorance of phenomenally-bound consciousness is architecturally hardwired. Classical digital computers are zombies or (if consciousness is fundamental to the world) micro-experiential zombies, not phenomenally-bound subjects of experience with a pleasure-pain axis. Speed of execution or complexity of code make no difference: phenomenal unity isn’t going to “switch on”. Digital minds are an oxymoron.
Like the poster, I worry about s-risks. I just don’t think this is one of them.
Could you possibly say a bit more about why the mirror test is inadequate as a test of possession of a self-concept? Either way, making self-awareness a precondition of moral status has troubling implications. For example, consider what happens to verbally competent adults when feelings intense fear turn into uncontrollable panic. In states of “blind” panic, reflective self-awareness and the capacity for any kind of meta-cognition is lost. Panic disorder is extraordinarily unpleasant. Are we to make the claim that such panic-ridden states aren’t themselves important—only the memories of such states that a traumatised subject reports when s/he regains a measure of composure and some semblance of reflective self-awareness is restored? A pig, for example, or a prelinguistic human toddler, doesn’t have the meta-cognitive capacity to self-reflect on such states. But I don’t think we are ethically entitled to induce them—any more than we are ethically entitled to waterboard a normal adult human. I would hope posthuman superintelligence can engineer such states out of existence - in human and nonhuman animals alike.
Eugine, in answer to your question: yes. If we are committed to the well-being of all sentience in our forward light-cone, then we can’t simultaneously conserve predators in their existing guise. (cf. http://www.abolitionist.com/reprogramming/index.html) Humans are not obligate carnivores; and the in vitro meat revolution may shortly make this debate redundant; but it’s questionable whether posthuman superintelligence committed to the well-being of all sentience could conserve humans in their existing guise either.
Eliezer, in my view, we don’t need to assume meta-ethical realism to recognise that it’s irrational—both epistemically irrational and instrumentally irrational—arbitrarily to privilege a weak preference over a strong preference. To be sure, millions of years of selection pressure means that the weak preference is often more readily accessible. In the here-and-now, weak-minded Jane wants a burger asap. But it’s irrational to confuse an epistemological limitation with a deep metaphysical truth. A precondition of rational action is understanding the world. If Jane is scientifically literate, then she’ll internalise Nagel’s “view from nowhere” and adopt the God’s-eye-view to which natural science aspires. She’ll recognise that all first-person facts are ontologically on a par—and accordingly act to satisfy the stronger preference over the weaker. So the ideal rational agent in our canonical normative decision theory will impartially choose the action with the highest expected utility—not the action with an extremely low expected utility. At the risk of labouring the obvious, the difference in hedonic tone induced by eating a hamburger and a veggieburger is minimal. By contrast, the ghastly experience of having one’s throat slit is exceptionally unpleasant. Building anthropocentric bias into normative decision theory is no more rational than building geocentric bias into physics.
Paperclippers? Perhaps let us consider the mechanism by which paperclips can take on supreme value. We understand, in principle at least, how to make paperclips seem intrinsically supremely valuable to biological minds—more valuable than the prospect of happiness in the abstract. [“Happiness is a very pretty thing to feel, but very dry to talk about.”—Jeremy Bentham]. Experimentally, perhaps we might use imprinting (recall Lorenz and his goslings), microelectrodes implanted in the reward and punishment centres, behavioural conditioning and ideological indoctrination—and perhaps the promise of 72 virgins in the afterlife for the faithful paperclipper. The result: a fanatical paperclip fetishist! Moreover, we have created a full-spectrum paperclip -fetishist. Our human paperclipper is endowed, not merely with some formal abstract utility function involving maximising the cosmic abundance of paperclips, but also first-person “raw feels” of pure paperclippiness. Sublime!
However, can we envisage a full-spectrum paperclipper superintelligence? This is more problematic. In organic robots at least, the neurological underpinnings of paperclip evangelism lie in neural projections from our paperclipper’s limbic pathways—crudely, from his pleasure and pain centres. If he’s intelligent, and certainly if he wants to convert the world into paperclips, our human paperclipper will need to unravel the molecular basis of the so-called “encephalisation of emotion”. The encephalisation of emotion helped drive the evolution of vertebrate intelligence—and also the paperclipper’s experimentally-induced paperclip fetish / appreciation of the overriding value of paperclips. Thus if we now functionally sever these limbic projections to his neocortex, or if we co-administer him a dopamine antagonist and a mu-opioid antagonist, then the paperclip-fetishist’s neocortical representations of paperclips will cease to seem intrinsically valuable or motivating. The scales fall from our poor paperclipper’s eyes! Paperclippiness, he realises, is in the eye of the beholder. By themselves, neocortical paperclip representations are motivationally inert. Paperclip representations can seem intrinsically valuable within a paperclipper’s world-simulation only in virtue of their rewarding opioidergic projections from his limbic system—the engine of phenomenal value. The seemingly mind-independent value of paperclips, part of the very fabric of the paperclipper’s reality, has been been unmasked as derivative. Critically, an intelligent and recursively self-improving paperclipper will come to realise the parasitic nature of the relationship between his paperclip experience and hedonic innervation: he’s not a naive direct realist about perception. In short, he’ll mature and acquire an understanding of basic neuroscience.
Now contrast this case of a curable paperclip-fetish with the experience of e.g. raw phenomenal agony or pure bliss—experiences not linked to any fetishised intentional object. Agony and bliss are not dependent for their subjective (dis)value on anything external to themselves. It’s not an open question (cf. http://en.wikipedia.org/wiki/Open-question_argument) whether one’s unbearable agony is subjectively disvaluable. For reasons we simply don’t understand, first-person states on the pleasure-pain axis have a normative aspect built into their very nature. If one is in agony or despair, the subjectively disvaluable nature of this agony or despair is built into the nature of the experience itself. To be panic-stricken, to take another example, is universally and inherently disvaluable to the subject whether one is a fish or a cow or a human being.
Why does such experience exist? Well, I could speculate and tell a naturalistic reductive story involving Strawsonian physicalism (cf. http://en.wikipedia.org/wiki/Physicalism#Strawsonian_physicalism) and possible solutions to the phenomenal binding problem (cf. http://cdn.preterhuman.net/texts/body_and_health/Neurology/Binding.pdf). But to do so here opens a fresh can of worms.
Eliezer, I understand you believe I’m guilty of confusing an idiosyncratic feature of my own mind with a universal architectural feature of all minds. Maybe so! As you say, this is a common error. But unless I’m ontologically special (which I very much doubt!) the pain-pleasure axis discloses the world’s inbuilt metric of (dis)value—and it’s a prerequisite of finding anything (dis)valuable at all.
“Health is a state of complete [sic] physical, mental and social well-being”: the World Health Organization definition of health. Knb, I don’t doubt that sometimes you’re right. But Is phasing out the biology of involuntary suffering really too “extreme”—any more than radical life-extension or radical intelligence-amplification? When talking to anyone new to transhumanism, I try also to make the most compelling case I can for radical superlongevity and extreme superintelligence—biological, Kurzweilian and MIRI conceptions alike. Yet for a large minority of people—stretching from Buddhists to wholly secular victims of chronic depression and chronic pain disorders—dealing with suffering in one guise or another is the central issue. Recall how for hundreds of millions of people in the world today, time hangs heavy—and the prospect of intelligence-amplification without improved subjective well-being leaves them cold. So your worry cuts both ways.
Anyhow, IMO the makers of the BIOPS video have done a fantastic job. Kudos. I gather future episodes of the series will tackle different conceptions of posthuman superintelligence—not least from the MIRI perspective.
SaidAchmiz, you’re right. The issue isn’t settled: I wish it were so. The Transhumanist Declaration (1998, 2009) of the World Transhumanist Association / Humanity Plus does express a non-anthropocentric commitment to the well-being of all sentience. [“We advocate the well-being of all sentience, including humans, non-human animals, and any future artificial intellects, modified life forms, or other intelligences to which technological and scientific advance may give rise” : http://humanityplus.org/philosophy/transhumanist-declaration/] But I wonder what percentage of lesswrongers would support such a far-reaching statement?
ArisKatsaris, it’s possible to be a meta-ethical anti-realist and still endorse a much richer conception of what understanding entails than mere formal modeling and prediction. For example, if you want to understand what it’s like to be a bat, then you want to know what the textures of echolocatory qualia are like. In fact, any cognitive agent that doesn’t understand the character of echolocatory qualia-space does not understand bat-minds. More radically, some of us want to understand qualia-spaces that have not been recruited by natural selection to play any information-signalling role at all.
shminux, Strawsonian physicalism may be false; but it is not dualism. Recall the title of Strawson’s controversial essay was “Realistic monism—why physicalism entails panpsychism” (Journal of Consciousness Studies 13 (10-11):3-31 (2006)) For an astute critique of Strawson, perhaps see William Seager’s “The ‘intrinsic nature’ argument for panpsychism” (Journal of Consciousness Studies 13 (10-11):129-145 (2006) http://philpapers.org/rec/SEATIN ) Once again, I’m not asking you to agree here. We just need to be wary of dismissing a philosophical position without understanding the arguments that motivate it.
It’s good to know we agree on genetically phasing out the biology of suffering!
Now for your thought-experiments.Quantitatively, given a choice between a tiny amount of suffering X + everyone and everything else being great, or everyone dying, NU’s would choose omnicide no matter how small X is?
To avoid status quo bias, imagine you are offered the chance to create a type-identical duplicate, New Omelas—again a blissful city of vast delights dependent on the torment of a single child. Would you accept or decline? As an NU, I’d say “no”—even though the child’s suffering is “trivial” compared to the immensity of pleasure to be gained. Likewise, I’d painlessly retire the original Omelas too. Needless to say, our existing world is a long way from Omelas. Indeed, if we include nonhuman animals, then our world may contain more suffering than happiness. Most nonhuman animals in Nature starve to death at a early age; and factory-farmed nonhumans suffer chronic distress. Maybe the CU should press a notional OFF button and retire life too.
A separate but related question: What if we also make it so that X doesn’t happen for sure, but rather happens with some probability. How low does that probability have to be before NUs would take the risk, instead of choosing omnicide? Is any probability too low?
You pose an interesting hypothetical that I’d never previously considered. If I could be 100% certain that NU is ethically correct, then the slightest risk of even trivial amounts of suffering is too high. However, prudence dictates epistemic humility. So I’d need to think some more before answering.
Back in the real world, I believe (on consequentialist NU grounds) that it’s best to enshrine in law the sanctity of human and nonhuman animal life. And (like you) I look forward to the day when we can get rid of suffering—and maybe forget NU ever existed.
drnickbone, the argument that meat-eating can be ethically justified if conditions of factory-farmed animals are improved so their lives are “barely” worth living is problematic. As it stands, the argument justifies human cannibalism. Breeding human babies for the pot is potentially ethically justified because the infants in question wouldn’t otherwise exist—although they are factory- farmed, runs this thought-experiment, their lives are at least “barely” worth living because they don’t self-mutilate or show the grosser signs of psychological trauma. No, I’m sure you don’t buy this argument—but then we shouldn’t buy it for nonhuman animals either.
Eliezer, is that the right way to do the maths? If a high-status opinion-former publicly signals that he’s quitting meat because it’s ethically indefensible, then others are more likely to follow suit—and the chain-reaction continues. For sure, studies purportedly showing longer lifespans, higher IQs etc of vegetarians aren’t very impressive because there are too many possible confounding variables. But what such studies surely do illustrate is that any health-benefits of meat-eating vs vegetarianism, if they exist, must be exceedingly subtle. Either way, practising friendliness towards cognitively humble lifeforms might not strike AI researchers as an urgent challenge now. But isn’t the task of ensuring that precisely such an outcome ensues from a hypothetical Intelligence Explosion right at the heart of MIRI’s mission—as I understand it at any rate?