asr

Karma: 1,674

asr 2 Jun 2011 4:39 UTC
8 points
on: Pluralistic Moral Reductionism
I feel like your austere meta-ethicist is mostly missing the point. It’s utterly routine for different people to have conflicting beliefs about whether a given act is moral*. And often they can have a useful discussion, at the end of which one or both participants change their beliefs. These conversations can happen without the participants changing their definitions of words like ‘moral’, and often without them having a clear definition at all.

[This is my first LW comment—if I do something wrong, please bear with me]

This suggests that precise definitions or agreement about definitions isn’t all that critical. But it’s sometimes useful to be able to reason from stipulated and mutually agreed definitions, in which case meta-ethical speculation and reasoning is doing useful work if it offers a menu of crisp, useful, definitions that can be used in discussion of specific moral claims. Relatedly, it’s also doing useful work by offering a set of definitions that help people conceptualize and articulate their personal feelings about morality, even absent a concrete first-order question.

And part of what goes into picking definitions is to understand their consequences. A philosopher is doing useful work for me if he shows me that a tempting-sounding definition of ‘morality’ doesn’t pick out the set of things I want it to pick out, or that some other definition turns out not to refer to any clear set at all.

Many mathematical entities have multiple logically equivalent definitions, that are of different utility in different contexts. (E.g., sometimes I want to think about a circle as a locus of points, and sometimes as the solution set to an equation.) In the real world, something similar happens.

When I discuss, say, abortion, with somebody, probably there are multiple working definitions of ‘moral’ that could be mutually agreed upon for the purpose of the conversation, and the underlying dispute would still be nontrivial and intelligible. But some definitions might be more directly applicable to the discussion—and philosophical reasoning might be helpful in figuring out what the consequences of various definitions are. For instance, a non-cognitive strikes me intuitively as less likely to be useful—but I’d be open to an argument showing how it could be useful in a debate.

Probably a great deal of academic writing on meta-ethics is low value. But that’s true of most writing on most topics and doesn’t show that the topic is pointless. (With academics being major offenders, but not the only offenders.)

*I’m thinking of the individual personal changes in belief that went along with increased opposition to official racism in America over the course of the 20th century. Or opposition to slavery in the 19th.

asr 2 Jun 2011 19:21 UTC
1 point
in reply to: Garren’s comment on: Pluralistic Moral Reductionism
I agree with all the claims you’re making about morality and about moral discussion. But I don’t quite see where any of this is giving me any new insights or tools. Sure, people have different but often overlapping values. I knew that. I think most adults who ever have conversations about morality know that. And we know that without worrying too much about the definition of morality and related words.

But I think everything you’ve said is also true about personal taste in non moral questions. I and my friends have different but overlapping taste in music, because we have distinct but overlapping set of desiderata for what we listen to. And sometimes, people get convinced to like something they previously didn’t. I want a meta-ethics that gives me some comparative advantage in dealing with moral problems, as compared to other sorts of disagreements. I had assumed that lukeprog was trying to say something specifically about morality, not just give a general and informal account of human motivation, values, and preferences.

Thus far, this sequence feels like a lot of buildup and groundwork that is true but mostly not in much dispute and mostly doesn’t seem to help me accomplish anything. Perhaps my previous comment should just have been a gentle nudge to lukeprog to get to the point.

asr 5 Jun 2011 8:33 UTC
2 points
in reply to: lukeprog’s comment on: Pluralistic Moral Reductionism
Having had time to mull over—I think here’s something about your post that bothers me. I don’t think it’s possible to pinpoint a single sentence, but here are two things that don’t quite satisfy me.

1) Neither your austere or empathetic meta-ethicists seem to be telling me anything I wanted to hear. What I want is a “linguistic meta-ethicist”, who will tell me what other competent speakers of English mean when they use “moral” and suchlike terms. I understand that different people mean different things, and I’m fine with an answer which comes in several parts, and with notes about which speakers are primarily using which of those possible definitions.

What I don’t want is a brain scan from each person I talk to—I want an explanation that’s short and accessible enough to be useful in conversations. Conventional ethics and meta-ethics has given a bunch of useful definitions. Saying “well, it depends” seems unnecessarily cautious; saying “let’s decode your brain” seems excessive for practical purposes.

2) Most of the conversations I’m in that involve terms like “moral” would be only slightly advanced by having explicit definitions—and often the straightforward terms to use instead of “moral” are very nearly as contentious or nebulous. In your own examples, you have your participants talk about “well-being” and “non-moral goodness.” I don’t think that’s a significant step forward. That’s just hiding morality inside the notion of “a good life”—which is a sensible thing to say, but people have been saying it since Plato, and it’s an approach that has problems of its own.

By the way, I do understand that I may not have been your target audience, and that the whole series of posts has been carefully phrased and well organized, and I appreciate that.

asr 5 Jun 2011 18:48 UTC
2 points
in reply to: lukeprog’s comment on: Pluralistic Moral Reductionism
I would find it helpful—and I think several of the other posters here would as well—to see one reduction on some nontrivial question carried far enough for us to see that the process can be made to work. If I understand right, your approach requires that speakers, or at least many speakers much of the time, can reduce from disputed, loaded, moral terms to reasonably well-defined and fact-based terminology. That’s the point I’d most like to see you spend your space budget on in future posts.

Definitions are good. Precise definitions are usually better than loose definitions. But I suspect that in this context, loose definitions are basically good enough and that there isn’t a lot of value to be extracted by increased precision there. I would like evidence that improving our definitions is a fruitful place to spend effort.

I did read your post on conceptual analysis. I just re-read it. And I’m not convinced that the practice of conceptual analysis is any more broken than most of what people get paid to do in the humanities and social sciences . My sense is that the standard textbook definitions are basically fine, and that the ongoing work in the field is mostly just people trying to get tenure and show off their cleverness.

I don’t see that there’s anything terribly wrong with the practice of conceptual analysis—so long as we don’t mistake an approximate and tentative linguistic exercise for access to any sort of deep truth.

asr 5 Jun 2011 22:33 UTC
0 points
in reply to: Garren’s comment on: Pluralistic Moral Reductionism
At risk of triggering the political mind-killer, I think there are some potentially problematic consequences of this view.

Once a disagreement is known to result from a pure difference in values, there isn’t a rational way to resolve it...the best we can do is make people aware of the difference in their claims.

Suppose we don’t have good grounds for keeping one set of moral beliefs over another. Now suppose somebody offers to reward us for changing our views, or punish us for not changing. Should we change our views?

To go from the philosophical to the concrete: There are people in the world who are fanatics who are largely committed to some reading of the Bible/Koran/Little Green Book of Colonel Gaddafi/juche ideology of the Great Leader/whatever. Some of those people have armies and nuclear weapons. They can bring quite a lot of pressure to bear on other individuals to change their views to resemble those of the fanatic.

If rationalism can’t supply powerful reasons to maintain a non-fanatical worldview in the face of pressure to self-modify, that’s an objection to rationalism. Conversely, altering the moral beliefs of fanatics with access to nuclear weapons strikes me as an extremely important practical project. I suspect similar considerations will apply if you consider powerful unfriendly powerful AIs.

This reminds me of that line of Yeats, that “the best lack all conviction, while the worst are full of passionate intensity.” Ideological differences sometimes culminate in wars, and if you want to win those wars, you may need something better than “we have our morals and they have theirs.”

To sharpen the point slightly: There’s an asymmetry between the rationalists and the fanatics, which is that the rationalists are aware that they don’t have a rational justification for their terminal values, but the fanatic does have a [fanatical] justification. Worse, the fanatic has a justification to taboo thinking about the problem, and the rationalist doesn’t.

asr 5 Jun 2011 23:00 UTC
2 points
in reply to: Peterdjones’s comment on: Pluralistic Moral Reductionism

It’s an analytical truth that you shouldn’t do to others what you wouldn’t want done to yourself.

I’m skeptical that this statement is true, let alone an analytic truth. Different people have different desires. I take the Golden Rule to be a valuable heuristic, but no more than that.

What is your reason for believing that it is true as an absolute rule?

asr 6 Jun 2011 0:07 UTC
2 points
in reply to: Manfred’s comment on: Pluralistic Moral Reductionism

Just because morality is personal doesn’t make it not real. If you model people as agents with utility functions, the reason not to change is obvious—if you change, you won’t do all the things you value.

Does this imply that if a rational actor has terminal values that are internally consistent and in principle satisfiable, it would always be irrational for the actor to change those values or allow them to change?

That doesn’t seem right either. Somehow, an individual improving their moral beliefs as they mature, the notional Vicar of Bray, and Pierre Laval are all substantially different cases of people changing their [terminal] beliefs in response to events. There’s something badly wrong with a theory that can’t distinguish those cases.

Also, my apologies if this has been already discussed to death on LW or elsewhere—I spent some time poking and didn’t see anything on this point.

asr 6 Jun 2011 0:46 UTC
2 points
on: Meanings of Mathematical Truths
There are elementary statements about arithmetic that don’t have obvious real-world equivalents, or where the obvious physical implication is false.

Suppose I tell you, for instance, that there’s a real number that when multiplied by itself equals 2. There’s a natural geometric interpretation of this in terms of ratios; it says that given two segments, A and B, there’s a segment C such that A:C is the same ratio as C:B.

But it could very easily be the case that this is not true in the actual geometry of the universe around us; measurements are noisy, space is curved, space isn’t quite continuous, etc. But the math is unobjectionable and exact.

So that suggests that at least some elementary math can be, and has to be, justified in some way other than “true for the universe around us.”

asr 6 Jun 2011 4:06 UTC
0 points
in reply to: Manfred’s comment on: Pluralistic Moral Reductionism
I would cheerfully agree that humans aren’t rational and routinely change their minds about morality for non-rational reasons.

This is one of the things I was trying to get at. Ask when we should change our minds for non-rational reasons, and when we should attempt to change others’ minds using non-rational means.

The same examples I mentioned above work for these questions too.

Here’s what I had in mind with the reference to the Vicar of Bray. Imagine an individual with two terminal values: “Stay alive and employed” and the reigning orthodoxy at the moment. The individual sincerely believes in both, and whenever they start to conflict, changes their beliefs about the orthodoxy. He is quite sincere in advocating for the ruling ideology at each point in time; he really does believe in divine right of kings, just so long as it’s not a dangerous belief to hold.

The beliefs in question are at least potentially terminal moral beliefs. Without delving deep into the history, let’s stipulate for the purpose of the conversation that we’re talking about a rational actor who has a sequence of terminal moral beliefs about what constitutes a just government, and that these beliefs shift with the political climate.

Now for contrast, let’s consider a hypothetical rational but very selfish child. The child’s parents attempt and succeed in changing the child’s values to be less selfish. They do this by the usual parental tactics of punishment and example-setting, not by rational argument. By your social standard and mine, this is an improvement to the child.

Both the vicar and the child are updating their moral beliefs in response to outside pressure, not rational deliberation. The general consensus is that parents are obligated to bring up their children not to be overly self-centered and that reasoning with children is not a sufficient pedagogic technique But conversely that coercive government pressure on religion is ignoble.

Is this simply that you and I think “a change in moral beliefs, brought about by non-reasonable means is good (all else equal), if it significantly improves the beliefs of the subject by my standards”?

asr 6 Jun 2011 4:12 UTC
2 points
in reply to: Vladimir_Nesov’s comment on: Meanings of Mathematical Truths

The only way in which we can access an abstract mathematical fact, learn about its truth, is through some physical tools, such that we believe their behavior to be linked to the abstract statement.

This has not been my experience in mathematics. There’s lots and lots of theorems that I know where I don’t have any physical counterpart for the claim. Cantor’s diagonalization argument, for instance. The irrationality of the square root of two. The fundamental theorem of arithmetic, aka the unique prime factorization theorem.

For natural-number arithmetic, geometry, and calculus, it’s often easy to find close physical analogs. For the rest of math, I don’t see it.

Are you using “physical tools” or “linked to” in a much broader sense than I understand?

asr 6 Jun 2011 7:21 UTC
0 points
in reply to: Garren’s comment on: Pluralistic Moral Reductionism

I think the worry here is that realizing ‘right’ and ‘wrong’ are relative to values might make us give up our values. Meanwhile, those who aren’t as reflective are able to hold more strongly onto their values.

Almost. What I’m worried about is that acknowledging or defining values to be arbitrary makes us less able to hold onto them and less able to convince others to adopt values that are safer for us. I think it’s nearly tautological that right and wrong are defined in terms of values.

The comment about fanatics with nuclear weapons wasn’t to indicate that that’s a particular nightmare of mine. It isn’t. Rather, that was to get at the point that moral philosophy isn’t simply an armchair exercise conducted amongst would-be rationalists—sometimes having a good theory a matter of life and death.

It’s very tempting, if you are firmly attached to your moral beliefs, and skeptical about your powers of rationality (as you should be!) to react to countervailing opinion by not listening. If you want to preserve the overall values of your society, and are skeptical of others’ powers of rational judgement, it’s tempting to have the heretic burnt at the stake, or the philosopher forced to drink the hemlock.

One of the undercurrents in the history of philosophy has been an effort to explain why a prudent society that doesn’t want to lose its moral footings can still allow dissent, including dissent about important values, that risks changing those values to something not obviously better. Philosophers, unsurprisingly, are drawn to philosophies that explain why they should be allowed to keep having their fun. And I think that’s a real and valuable goal that we shouldn’t lose sight of.

I’m willing to sacrifice a bunch of other theoretical properties to hang on to a moral philosophy that explains why we don’t need heresy trials and why nobody needs to bomb us for being infidels.

asr 7 Jun 2011 7:36 UTC
0 points
in reply to: prase’s comment on: Meanings of Mathematical Truths
In the actual physical world we live in, statements like “there is a square of area two” might not be exactly true. There certainly is no evidence that they are exactly true. (Whereas, in contrast, two plus two really gives you exactly four bananas.)

It’s certainly true that much human-developed math was developed to serve practical purposes, and therefore does accurately model aspects of the real world. But the math isn’t made less true because the physical world deviates slightly from it; likewise the math that’s less tied to the physical world isn’t less true. There are lots and lots of theorems that are interesting, and even useful, but that don’t seem to have much to do with anything physical. (E.g., number theory, or abstract algebra.)

asr 7 Jun 2011 7:53 UTC
1 point
in reply to: Manfred’s comment on: Pluralistic Moral Reductionism
I think the caveats will turn out to matter a lot. One of the things that human moral beliefs do, in practice, is give other humans some reasons to trust you. If I know that you are committed, for non-instrumental reasons, to avoid manipulating* me into changing my values, that gives me reasons to trust you. Conversely, if your moral view is that it’s legitimate to lie to people to make them do what you want, people will trust you less.

Obviously, people have incentives to lie about their true values. I think equally obviously, people are paying attention and looking hard for that sort of hypocrisy.

*This sentence is true for a range of possible expansions of “manipulating”.

asr 7 Jun 2011 14:58 UTC
0 points
in reply to: prase’s comment on: Meanings of Mathematical Truths
In this case, I mean something like “if you start off with consistent and true beliefs, adding more true beliefs won’t lead to self contradiction.” I can define self-contradiction formally, as asserting both a statement and its formal negation.

This may seem slightly circular, but I think it’s still a useful definition that captures what I want. I also think some circularity is useful to capture what we mean by an axiomatic system.

asr 7 Jun 2011 15:26 UTC
2 points
on: St. Petersburg Mugging Implies You Have Bounded Utility
My poor feeble meat brain can only represent finitely many numbers. My subjective probability that you’ll pay off on the bet rounds to zero as the utilities get big. So the sum converges to less than a dollar, even without hitting an upper bound on utilities.

asr 7 Jun 2011 19:19 UTC
3 points
in reply to: TimFreeman’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
Yes, certainly. I wasn’t supplying evidence against the “bounded utility” conclusion, just suggesting that there are alternate interpretations under which bounded utility doesn’t come up.

Having thought about it, I’m pretty sure that my preferences can be modeled with bounded utilities. I would justify that conclusion using the fact that there are only finitely many distinguishable worlds or mental states. If each of those has a finite utility, then my overall utility function is bounded.

asr 10 Jun 2011 7:09 UTC
3 points
in reply to: lukeprog’s comment on: Pluralistic Moral Reductionism
I agree that emotivism is an accurate description, much of the time, for what people mean when they make value judgments. I would also agree that most people don’t have a specific or precise definition in mind. But emotivism isn’t the only description and for practical purposes it’s often not the most useful. Among other things, we have to specify which emotion we are talking about. Not all disgust is moral disgust.

Value judgments show up routinely in law and in daily life. It would be an enormous, difficult, and probably low-value task to rewrite our legal code to avoid terms like “good cause”, “unjust enrichment”, “unconscionable contract”, and the like. Given that we’re stuck with moral language, it’s a useful project to pull out some definitions to help focus discourse slightly. But we aren’t going to be able to eliminate them. “Morality” and its cousins are too expensive to taboo.

We want law and social standards to be somewhat loosely defined, to avoid unscrupulous actors trying to worm their way through loopholes. We don’t want to be overly precise and narrow in our definitions—we want to leverage the judgement of judges and juries. But conversely, we do want to give them guidance about what we mean by those words. And precedent supplies one sort of guidance, and some definitions give them an additional sort of guidance.

I suspect it would be quite hard to pick out precisely what we as a society mean when we use those terms in the legal code—and very hard to reduce them to any sort of concrete physical description that would still be human-intelligible. I would be interested to see a counterexample if you can supply one easily.

I have the sense that trying to talk about human judgement and society without moral language would be about like trying to discuss computer science purely in terms of the hardware—possible, but unnecessarily cumbersome.

One of the common pathologies of the academy is that somebody comes up with a bright idea or a powerful intellectual tool. Researchers then spend several years applying that tool to increasingly diverse contexts, often where the marginal return from the tool is near-zero. Just because conceptual analysis is being over-used doesn’t mean that it is always useless! The first few uses of it may indeed have been fairly high-value in aiding us in communicating. The fact that the tool is then overused isn’t a reason to ignore it.

Endless wrangles about definitions, I think are necessarily low-value. Working out a few useful definitions or explanations for a common term can be valuable, though—particularly if we are going to apply those terms in a quasi-formal setting, like law.

asr 10 Jun 2011 7:12 UTC
0 points
in reply to: NancyLebovitz’s comment on: Safety Culture and the Marginal Effect of a Dollar
Yes. As tim points out below, the main thing that programmers are taught is “self-modifying programs are almost always more trouble than they’re worth—don’t do it.”

My hunch is that self-modifying AI is far more likely to crash than it is to go FOOM, and that non-self-modifying AI (or AI that self-modifies in very limited ways) may do fairly well by comparison.

What would defuse unfriendly AI?

asr10 Jun 2011 7:27 UTC

7 points

14 comments1 min readLW link

asr 10 Jun 2011 7:59 UTC
0 points
in reply to: Zetetic’s comment on: Safety Culture and the Marginal Effect of a Dollar
My comment wasn’t directed towards CEV at all—CEV sounds like a sensible working definition of “friendly enough”, and I agree that it’s probably computationally hard.

I was suggesting that any program, AI or no, that is coded to rewrite critical parts of itself in substantial ways is likely to go “splat”, not “FOOM”—to degenerate into something that doesn’t work at all.

asr

What would de­fuse un­friendly AI?

What would defuse unfriendly AI?