[SEQ RERUN] Unnatural Categories
Today’s post, Unnatural Categories was originally published on 24 August 2008. A summary (taken from the LW wiki):
There are some mental categories we draw that are relatively simple and straightforward. Others get trickier, because they are primarily drawn in such a way that whether or not something fits into that category is important information to our utility function. Deciding whether someone is “alive”, for instance. Is someone like Terry Schaivo alive? This issue is why, in part, technology creates new moral dilemmas, and why teaching morality to a computer is so hard.
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we’ll be going through Eliezer Yudkowsky’s old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Mirrors and Paintings, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day’s sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
This is a tangent, but the long discussion of poisons reminded me of my recent discovery of the (not surprising in retrospect) limitations in our knowledge. In Plato scholarship, there was for some time a controversy about Socrates’ death scene in Phaedo; some scholars argued that death by hemlock is much more unpleasant than the peaceful scene Plato describes, so Plato either got it wrong or misrepresented it for some purpose. But more recent scholars mostly think that Plato probably did describe it accurately, with the alternate interpretation arising because quite a number of different things have been called “hemlock” over the years, some of which produce very unpleasant deaths, but at least one of which produces pretty much what Plato describes. It appears that one reason that there was ever a controversy is that we often don’t know very much about how poisons cause deaths; once something is identified as poisonous, people just start avoiding it, so we don’t tend to accumulate many further cases of death by that cause to observe.
Except, presumably, in cases where people deliberately use those poisons as means of execution, murder or warfare?
The last of these caused a major extinction event...
And there’s a famous myth that the second did so as well...
Not by being poison, though.
The post seems a long way of saying something I took from Stirner—categorize according to your values, instead of feeling compelled to value according to your categories.
But then you have to waste computing power carrying around two sets of categories.
Is anyone else bothered by the apparent dichotomy into “natural” and “unnatural” categories? There’s room for a matter of degree here—and that’s probably all the argument ultimately needs—but the terminology is binary.
Suppose there is at least one utility function, and an agent who has it. Then all categories the agent ultimately finds useful are ipso facto “constructed [...] in an observable, testable way from categories themselves simple”. Namely, from the categories “this agent” and “utility function”. And (I think—objections welcome) you don’t have to share this agent’s values to construct it.
Is there any physical object or property that absolutely must be used (not merely recognizable in principle, but actually used) in the ontology of all conceivable rational beings, no matter their values? I doubt it. If that is the standard of “natural category” then all categories are unnatural. But if it is enough that the category be recognizable in principle, then (see the paragraph above) all categories are natural.