Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very conceptcould be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
First, EY makes it abundantlyclear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY makes it abundantly clear that two agents can have a fundamental disagreement on values
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
The core intuition is just that it’s possible to have irresolvable moral disagreements.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.
Thanks for the reply—I find this a very interesting topic. One thing I should clarify is that my view doesn’t entail giving aliens “veto powers”, as you put it; an alternative response is to take them to be unreasonable to intrinsically desire the eating of babies. That isn’t an intrinsically desirable outcome (I take it), i.e. there is no reason to desire such a thing. Stronger still, we may think it intrinsically undesirable, so that insofar as an agent has such desires they are contrary to reason. (This requires a substantive notion of reason that goes beyond mere instrumental rationality, of course.)
In any case, I’d put the crux of my argument slightly differently. The core intuition is just that it’s possible to have irresolvable moral disagreements. We can imagine a case where Bob is stubbornly opposed to abortion, and Jane is just as stubbornly in favour of it, and neither agent is disposed to change their mind in light of any additional information. EY’s view would seem to imply that the two agents mustn’t really disagree. And that just seems a mistake: it’s part of our concept of morality that this very concept could be shared by someone who fundamentally (and irresolvably) disagrees with us about what the substantive moral facts are. This is because we’re aspiring to conform our judgments to a standard that is outside of ourselves. (If you don’t think there are any such objective standards, then that’s just to say that there are no normative facts, given my concept of normativity.)
Richard, hello.
Human beings are analogous to computers. Morality and other aspects of behavior and cognition are analogous to programs. It is a type error to ask whether a program “really exists” somewhere outside a computer, or is “intrinsic” to a computer, or is “contingent”, or something like that. Such questions don’t correspond to observations within the world that could turn out one way or the other. You see a computer running a certain program and that’s the end of the story.
Your mind is a program too, and your moral intuitions are how your algorithm feels from inside, not a direct perception of external reality (human beings are physically incapable of that kind of thing, though they may feel otherwise). I know for a fact that you have no astral gate in your head to pull answers from the mysterious source of morality. But this doesn’t imply that your moral intuitions “should” be worthless to you and you “should” seek external authority! There’s nothing wrong with mankind living by its internal moral lights.
Yes, it’s possible that different computers will have different programs. Our world contains billions of similar “moist robots” running similar programs, perhaps because we were all created from design documents that are 99% identical for historical reasons, and also because we influence each other a lot. Your intuition that all “possible” sentient agents must share a common morality is unlikely to survive an encounter with any sentient agent that’s substantially different from a human. We can imagine such agents easily, e.g. a machine that will search for proofs to Goldbach’s conjecture and turn surrounding matter and energy into computing machinery to that end. Such a machine may be more ingenious than any human in creating other machines, discovering new physics, etc., but will never gravitate toward your intuition that one shouldn’t kill babies. Most possible “intelligent agents” (aka algorithms that can hit small targets in large search spaces) aren’t humans in funny suits.
I expect Richard’s memeset has understanding of all your points that doesn’t move his current position. You’re probably exposing him to arguments he has already encountered, so there’s little point in expecting a different result. I’m not saying that Richard can’t be moved by argument, just not by standard argument that is already known to have failed to move him. He even probably “agrees” with a lot of your points, just with a different and more sophisticated understanding than yours.
On the other hand, it might work for the benefit of more naive onlookers.
The intent of my comment wasn’t to convince Richard (I never do that), but to sharpen our points and make him clarify whatever genuine insight he possesses and we don’t.
That’s a motivation I didn’t consider. (Agreed.)
Yeah, as Vladimir guessed, this is all familiar.
Your last paragraph suggests that you’ve misunderstood my view. I’m not making an empirical claim to the effect that all agents will eventually converge to our values—I agree that that’s obviously false. I don’t even think that all formally intelligent agents are guaranteed to have normative concepts like ‘ought’, ‘reason’, or ‘morality’. The claim is just that such a radically different agent could share our normative concepts (in particular, our aspiration to a mind-independent standard), even if they would radically disagree with us about which things fall under the concept. We could both have full empirical knowledge about our own and each other’s desires/dispositions, and yet one (or both) of us might be wrong about what we really have reason to want and to do.
(Aside: the further claim about “reasons” in your last sentence presupposes a subjectivist view about reasons that I reject.)
What use is this concept of “reasonability”? Let’s say I build an agent that wants to write the first 1000 Fibonacci numbers in mile-high digits on the Moon, except skipping the 137th one. When you start explaining to the agent that it’s an “arbitrary omission” and it “should” amend its desires for greater “consistency”, the agent just waves you off because listening to you isn’t likely to further its current goals. Listening to you is not rational for the agent in the sense that most people on LW use the term: it doesn’t increase expected utility. If by “rational” you mean something else, I’d like to understand what exactly.
I mean ‘rational’ in the ordinary, indefinable sense, whereby calling a decision ‘irrational’ expresses a distinctive kind of criticism—similar to that expressed by the words ‘crazy’, ‘foolish’, ‘unwise’, etc. (By contrast, you can just say “maximizes expected utility” if you really mean nothing more than maximizes expected utility—but note that that’s a merely descriptive concept, not a normative one.)
If you don’t possess this concept—if you never have thoughts about what’s rational, over and above just what maximizes expected utility—then I can’t help you.
I don’t think we can make progress with such imprecise thinking. Eliezer has a nice post about that.
When we’re trying to reduce intuitions, there’s no helping starting from informal ideas. Another question is that it’s not proper to stop there, but Richard doesn’t exactly suggest that.
A more salient to me question is, What are suggestions about redefining this “rational” intuitive idea good for, if it’s supposed to be the source material? Such question even explains how the idea of considering, say, “consciousness”, in a more precise sense is methodologically a step in the wrong direction: when words are the data you work with, you should be careful to assign new words to new ideas used for analyzing them.
In this old comment, he does seem to suggest stopping there.
I’m not sure I understand your second paragraph. Are you suggesting that if we come up with a new theory to explain some aspect of consciousness, we should use a word other than “consciousness” in that theory, to avoid potentially losing some of our intuitions about consciousness?
Yes.
First, EY makes it abundantly clear that two agents can have a fundamental disagreement on values– it’s just not the best (or most helpful) assumption when you’re talking about two sane human beings with a vast sea of common frameworks and heuristics.
Secondly, I’m worried about what you’re trying to do with words when you suggest we “take them to be unreasonable to intrinsically desire the eating of babies”.
If you’re making an empirical claim that an alien with fundamentally different terminal values will (say) be uninterested in negotiating mutually beneficial deals, or will make patently suboptimal decisions by its own criteria, or exhibit some other characteristic of what we mean by “unreasonable”, then you’d need some strong evidence for that claim.
If instead you openly redefine “reasonable” to include “shares our fundamental moral standards”, then the property
becomes a tautology which no longer excludes “meta-semantic subjectivism”, as you put it. So I’m puzzled what you mean.
Talking past each other a bit here. Let me try again.
EY allows for disagreement in attitude: you might want one thing, while the babyeaters want something different. Of course I’m not charging him with being unable to accommodate this. The objection is instead that he’s unable to accommodate disagreement in moral judgment (at the fundamental level). Normativity as mere semantics, and all that.
Your second point rests on a false dichotomy. I’m not making an empirical claim, but nor am I merely defining the word “reasonable”. Rather, I’m making a substantive normative (non-empirical) hypothesis about which things are reasonable. If you can’t make sense of the idea of a substantive non-empirical issue, you may have fallen victim to scientism.
What is the difference between an in-principle irresolvable disagreement (moral or otherwise), and talking past each other (i.e. talking of different subject matters, or from different argument-processing frameworks)?
What fact have you established by manipulating the definition of a word in this manner? I want a meta-ethical theory that at least describes baby-eaters, because I don’t expect to have object-level understanding of human morality that is substantially more accurate than what you’d get if you add baby-eating impulses to it.