There’s an argument in the metaethics sequence, to the effect that there are no universally compelling moral arguments. This argument seems to be an important cashed thought (in don’t mean that in any pejorative sense) in LW discussions of morality. This argument also seems to me to be faulty. Can anyone help me see what I’m missing?
Yesterday, I proposed that you should resist the temptation to generalize over all of mind design space. If we restrict ourselves to minds specifiable in a trillion bits or less, then each universal generalization “All minds m: X(m)” has two to the trillionth chances to be false, while each existential generalization “Exists mind m: X(m)” has two to the trillionth chances to be true.
This would seem to argue that for every argument A, howsoever convincing it may seem to us, there exists at least one possible mind that doesn’t buy it.
The central inference in the argument seems to me to go like this:
P1) Any universal generalization over minds (‘All minds m: X(m)’) is very unlikely to be true.
P2) A purportedly universally compelling moral argument has the form ‘All minds m: X(m)’
C) A purportedly universally compelling moral argument is very unlikely to be true.
The reason I think this is faulty is that P1 is itself an argument of the form ‘All minds m: X(m)’, that is, it’s a universal generalization over minds. If that’s so, then P1 is very unlikely to be true, and we shouldn’t accept the argument. In order to save the argument, we would have to weaken P1 to cover a more specific set of generalizations over minds (so that P1 itself is excluded) but if we do this, then the argument is invalid, since universally compelling moral arguments may end up excluded as well. We might have good reasons for thinking they won’t be, but no such reasons are given in the sequence post.
I see the symmetry between P1 and a universally compelling moral argument in this: they both make a claim about the application of an argument quantifying over all minds in mind-space.
The claim EY is refuting is ‘For all minds m, m: (moral argument X is compelling)m.’
P1 makes the claim ‘For all minds m, m:(an argument of the form ‘for all minds m:X(m) is unlikely to be true)m.’
Thanks, you’re right that this isn’t self refuting. But with that P1, the argument seems invalid:
P1: For most predicates X: Not (For all minds m: X(m))
P2: UCMAs are X
C: Not UMCA
is like
P1: For most prime numbers n: (odd)n
P2: 2 is prime
C: 2 is odd
Edit: you might think that the conclusion is not that not ‘not UMCA’ but ‘UMCA is unlikely’, but this doesn’t follow either. I don’t know quite how ‘most’ quantifiers work, but I don’t think we can read a probabilistic conclusion off of them. I don’t think it follows from the above, for example, that 2 is likely to be odd.
Yes, the crucial issue in this conversation is the concept of ‘most’ and ‘probability’. What you can conclude from P1 is that a priori, a randomly selected predicate X probably does not satisfy X(m) for all m. If we had other reasons to believe that X(m) for all m, then we can update our beliefs. Similarly, we expect that a randomly selected prime number n is probably odd; but if we learn the further fact that n=2, then our belief changes.
So what do you make of this argument then? Suppose I were of the opinion that 2 is an even prime. You come to me with an argument to the effect that I should not believe 2 to be prime because a randomly selected prime number is very, very unlikely to be even. Should I be convinced by that? I may be convinced that in some sense, 2 is unlikely to be even, but I don’t think I should accept that 2 is not even, or that the evenness of 2 is questionable.
Similarly, suppose someone believes an argument to be universally compelling. It seems to me that EY’s argument should be unmoving: granting that it is unlikely for a randomly selected argument to be UC, but theirs is no randomly selected argument. And on DaFranker’s reading of this argument, the thesis that a given X is unlikely to hold for of all minds relies on the assumption that for most X’s, there is (something like) a 50% chance of its being true of some mind. But certainly a UCMAist won’t accept that this is true of UCMA’s. UCMA’s, they will say, are exactly those X’s for which this is not true.
The burden may be on them to justify the possibility of such an X, but that fact won’t save the argument.
As for your first paragraph, well, this is a straightforward application of Bayes’ theorem. If you’re sure that 2 is even, then learning that 2 was randomly selected from some distribution over primes should not be enough to change your credence very much.
As for your second and third paragraphs: Yes, the argument of Eliezer you’re talking about doesn’t refute the existence of universally compelling arguments; it merely means that you shouldn’t believe you have a universally compelling argument unless you have a good reason for believing so. If you think you have a good reason, then you don’t have to worry about this argument.
There’s a very simple argument refuting the existence of universally compelling arguments, and I believe it was stated elsewhere in this thread. It’s that argument you have to refute, not this one.
There’s a very simple argument refuting the existence of universally compelling arguments, and I believe it was stated elsewhere in this thread. It’s that argument you have to refute, not this one.
Please point this out to me if you get a chance, as I haven’t noticed it. And thanks for the discussion. I mean that: I can see that this wasn’t helpful or interesting for you, but rest assured it was for me, so your indulgence is appreciated.
You’re welcome! The refutation of universally compelling arguments I was referring to is this one. I see you responded that you’re interested in a different definition of “compelling”. On the word “compelling”, you say
On the one hand, we could mean ‘persuasive’ where this means something like ‘If I sat down with someone, and presented the moral argument to them, they would end up accepting it regardless of their starting view’. This seems to be a bad option, because the claim that ‘there are no universally persuasive moral arguments’ is trivial.
This is indeed the meaning of “compelling” that Eliezer uses, and Eliezer’s original argument is indeed trivial, which perhaps explains why he spent so few words on it.
If you wanted to defend a different claim, that there are arguments that all minds are “rationally committed” to accepting or whatever, then you’d have to begin by operationalizing “committed”, “reasons”, etc. I believe there’s no nontrivial way to do this. In any case the burden is on others to operationalize these concepts in an interesting way.
That I can’t argue with, though it wouldn’t follow from that that UCMAs are likely to be false.
EDIT: you edited your post, and so my reply doesn’t seem to make sense. In answer to your new question, I would say ‘I don’t, I just want some presentation of the argument on which its validity (or invalidity) is obvious’.
UCMA is making a claim about all minds, P1 is making a claim about some undefined subset of all minds.
They both talk about “all minds,” but only one of them makes a claim -about- all minds.
A parallel pair of arguments might be:
All squares are rectangles
The claim that all squares are rectangles is unlikely to be true of all squares.
The first claim is stronger than the second, and requires more proof. The fact that we can in fact prove it is irrelevant, and part of why I chose this example; consider the inverse propositions that all rectangles are squares, and that that claim is unlikely to be true, to see why this is important.
The claim that all squares are rectangles is unlikely to be true of all squares.
This is analogous to the conclusion of the above argument, not P1. An analogue to P1 would have to be something like ‘Any argument of the form ‘for all squares s:(X)s is unlikely to be true.’ The question would then be this: does this analogue of P1 count as an argument of the form ‘s:(X)s’? That is, does it quantify over all squares?
You might think it doesn’t, since it just talks about arguments. But my point isn’t quite that it must count as such an argument, but rather that it must count as an argument of the same form as P2 (whatever that might be). The reason is that P2 is not like ‘all squares are rectangles’. If it were, P2 would be a (purportedly) universally compelling moral argument. But P2 is rather the claim that there is such an argument. P2 is ‘for all minds m:(Moral Argument X is compelling)m’.
I see what you’re talking about. My confusion originates in your definition of P2, rather than P1, where I thought the confusion was originated.
Suppose two minds, A and B. A has some function for determining truth, let’s call it T. Mind B, on the other hand, is running an emulation of mind A, and its truth function is not(T).
Okay, yes, this is an utterly pedantic kind of argument, but I think it demonstrates that in -all- of mindspace, it’s impossible to have any universally compelling argument, without relying on balancing two infinities (number of possible arguments and number of possible minds) against each other and declaring a winner.
That sounds pretty good to me, though I think it’s an open question whether or not what you’re talking about is possible. That is, a UCMA theorist would accuse you of begging the question if you assumed at the outset that the above is a possibility.
It’s only an open question insofar as what are considered “minds” and “arguments” remain shrouded in mystery.
I’m rather certain that for a non-negligible fraction of all minds, the entire concept of “arguments” is nonsensical. There is, after all, no possible combination of inputs (or “arguments”), that will make the function “SomeMind(): Print 3” output that it is immoral to tube-feed chicken.
Because of my experience with programming and working with computation, I find it extremely unlikely that, out of all possible things, the specific way humans conceptualize persuasion and arguments would be a necessary requirement for any “mind” (which I take here as a ‘sentient’ algorithm in the largest sense) to function.
If the way we process these things called “arguments” is not a requirement for a mind, then there almost certainly exists at least one logically-possible mind design which does not have this way of processing things we call “arguments”.
As another intuition, if we adopt the Occam/Solomonoff philosophy for what is required to have a “mind”, then something as complicated as the process of understanding arguments, being affected, influenced or persuaded by them, by running through filters and comparing with prior knowledge and so on until some arguments convince or do not convince… that must be required for all possible minds as a component of an already-complex system called “minds”… sounds extremely much less common in the realm of all possible universes than the universes where simpler minds exist that do not have this property of understanding arguments and being moved by them.
I find it extremely unlikely that, out of all possible things, the specific way humans conceptualize persuasion and arguments would be a necessary requirement for any “mind” (which I take here as a ‘sentient’ algorithm in the largest sense) to function.
I don’t have any experience with programming at all, and that may be the problem: I just don’t see these reasons. To my mind (ha) a mind incapable of processing arguments, which is to say holding reasons in rational relations to each other or connecting premises and conclusions up in justificatory relations or whatever, isn’t reasonably called a mind. This may just be a failure of imagination on my part So...
As another intuition, if we adopt the Occam/Solomonoff philosophy for what is required to have a “mind”
Could you explain this? I’m under the impression that being capable of solomonoff induction requires being capable of 1) holding beliefs, 2) making inferences about those beliefs, 3) changing beliefs. Yet this seems to me to be all that is required for ‘understanding and being convinced by an argument’.
In my limited experience, UCMA supporters explicitly rejected the assertion that “arguments” and “being convinced by an argument” are equivalent to “evidence” and “performing a bayesian update on evidence”. So those three would be enough for evidence and updates, but not enough for argument and persuasion according to my next-best-guess of what they mean by “argument” and “convinced”.
For one, you need some kind of input system, and some process that looks at this input and connects it to pieces of an internal model, which requires and internal model and some structure that sends signals from the input to the process, and some structure where the process has modification access to other parts of the mind (to form the connections and perform the edits) in some way.
Then you need something that represents beliefs, and some weighing or filtering system where the elements of the input are judged (compared to other nodes in the current beliefs) and then evaluated using a bunch of built-in or learned rules (which implies having some rules of logic built-in to the structure of the mind, or the ability to learn such rules, both of which are non-trivial complexity-wise), and then those evaluations organized in a way where it can be concluded whether the argument is sound or not, and the previous judgments of the elements integrated so that it can be concluded whether the premises are also good, and then the mind also requires this result to send a signal to some dynamic process in the brain that modus ponens the whole thing into using the links to the concepts and beliefs to update and edit them to the new values prescribed by the compelling argument.
Whew, that’s a lot of stuff that we need to design into our mind that seems completely unnecessary for a mind to have sentience, as far as I can tell. I sure hope we don’t live in the kind of weird universes where sentience necessarily implies or requires all of the above!
Which is where the Occam/SI comes in. All of the above is weird, very specific, and extremely complex in most machine designs I can think of. Sentience is itself complex, but doesn’t seem to require the above as far as we can tell. Positing that minds also require all these additional complexities seems like a very bad idea. Statistically, ‘A’ is always more likely than ‘A and B’. Positing UCMA is a bit akin to positing ‘A and B and C and Fk but not Re and not any of Ke through Kz and L1..273 except L22’.
In my limited experience, UCMA supporters explicitly rejected the assertion that “arguments” and “being convinced by an argument” are equivalent to “evidence” and “performing a bayesian update on evidence”.
Eh, for the UCMA arguments I’m familiar with, they would be happy to work within the (excellent) Solomonoff framework as long as you allowed for probabilities of 0 and 1. I get that this isn’t an unproblematic allowance, but nothing about the math actually requires us to exclude probabilities of 0 and 1 (so far as I understand it).
Whew, that’s a lot of stuff that we need to design into our mind that seems completely unnecessary for a mind to have sentience, as far as I can tell.
What is necessary? It’ll pay off for us to get this on the table.
What is necessary? It’ll pay off for us to get this on the table.
If we knew exactly, someone would have a nobel for it and the nonperson predicate would be a solved problem by now, along with the Hard Problem of Consciousness and a throng of other things currently puzzling scientists the world over.
However, we do have a general idea of the direction to take, with an example here of some of the things involved. There’s still the whole debate and questions around the so-called “hard problem of consciousness”, but overall it doesn’t even seem as if the ability to communicate is required for consciousness or sentience, let alone hold the ability to parse language in a form remotely close to ours or that allows anything akin to an argument as humans are used to the word.
But past that point, the argument is no longer about UCMAs, and becomes about morality engines (and whether morality or something akin to it must exist in all minds), consciousness, what constitutes an ‘argument’ and ‘being convinced’, and other things humans yet understand so very little about.
Okay, I see the problem. Let’s say this: within the whole of mind-space there is a subset of minds capable of morally-evaluable behavior. For all such minds, the UCMA is true. This may be a tiny fraction, but the UCMAist won’t be disturbed by that: no UCMAist would insist that the UCMA is UC for minds incapable of anything relevant to morality. How does that sound?
This sounds like a good way to avoid the heavyweight problems with all the consciousness debates, so it seems like a good idea.
However, it retains the problem of defining “morality”, which is still unresolved. UCMAists will argue from theories of morality where UC is an element of the theory, while E.Y. already assumes a different metaethics where there is no clear boundaries of human “morality” and where morality-in-the-way-we-understand-it is a feature of humans exclusively, and other things might have things akin to morality that are not morality, and some minds would be able to evaluate moral behaviors without caring about morality in the slightest, while some other minds we might consider morally-important and yet would completely ignore any “UCMA” that would otherwise compel any human.
Without going into the details, you could hypothesize a simple mind than automatically rejects any argument. This would by itself prove the No Universally Compelling Arguments theory.
That would do it, though it may only attack a straw man: the thesis that the, say, categorical imperative is universally compelling is not the thesis that the CI is universally persuasive. Rather, I think the thought is that we are all rationally committed to the CI, whether we know or admit this or not.
Taboo compelling and restate. If compelling does not mean persuasive then what does it mean to you? Also taboo “committed” and “rational”—I think there’s a namespace conflict over your use of rational and the common Less Wrong usage, so restate using different terms. As a hint, try and imagine what a universally compelling argument would look like. What properties does it have? How do different minds react to understanding it, assuming they are capable of doing so? For bonus points explain what it means to be rationally committed to something (without using those words or synonyms).
Also worth noting: P1 is a generalization over statements about minds, not minds.
Well, we have two options in tabooing ‘compelling’. On the one hand, we could mean ‘persuasive’ where this means something like ‘If I sat down with someone, and presented the moral argument to them, they would end up accepting it regardless of their starting view’. This seems to be a bad option, because the claim that ‘there are no universally persuasive moral arguments’ is trivial. No one (of significance) has ever held the contrary view.
So our other option is to take ‘compelling’ as something like what Kantians say about the CI, namely that every mind is committed to it, whether they accept this or not (‘not’ out of irrationality). As you say, this leaves us with a lot more tabooing and explaining to do. I’m happy to go on with this, since it’s the sort of thing I enjoy, but it is a digression from my (perhaps confused) complaint about EY’s argument. The important bit there is just that ‘compelling’ probably shouldn’t be taken in such a way as to make EY’s point trivial.
The problem here is that the second option you offer does nothing to explain what a compelling argument is; it just passes the recursive buck onto the word “committed”. I know you said you recognize that, but unless we can show that this line of reasoning is coherent (let alone leads to a relevant conclusion, let alone correct) then there’s no reason to assume that Eliezer’s point isn’t trivial in the end. Philosophers have believed a lot of silly things, after all. The only sensible resolution I can come up with is where you take “committed to x” to mean “would, on reflection and given sufficient (accurate) information and a great deal more intelligence, believe x”. The problem is that this is still trivially false in the entirety of mindspace. You might, although I doubt it, be able to establish a statement of that form over all humans (I think Eliezer disagrees with me on the likelihood here). You could certainly not establish one given a mindspace that includes both humans and paper clip maximizers.
I know you said you recognize that, but unless we can show that this line of reasoning is coherent (let alone leads to a relevant conclusion, let alone correct) then there’s no reason to assume that Eliezer’s point isn’t trivial in the end.
If what you’re saying is this, then we agree: EY doesn’t here present an argument that UCMAs are likely to be false, but he does successfully argue that a certain class of generalizations over mind-space are likely to be false (such as generalizations about what minds will find persuasive) along with the assumption that a UCMA will fall into that class.
If that’s the line, then I think the argument is sound so far as it goes. UCMA enthusiasts (I am not among them, but I know them well) will not accept the final assumption, but you may be right that the burden is on them to show that UCMA’s (whatever ‘compelling’ is supposed to mean) does not fall into this class.
Alternatively, we could just posit that we’re only arguing against those people who do accept the assumption, that is those people who do take ‘compelling’ in UCMA to mean something like ‘immediately persuasive’, but then we’re probably tilting at windmills.
I suspect that our beliefs are close enough to each other at this point that any perceived differences are as likely to be due to minor linguistic quibbles as to actual disagreement. Which is to say, I wouldn’t have phrased it like you did (had I said it with that phrasing I would disagree) but I think that our maps are closer than our wording would suggest.
If anyone who does think they have a coherent definition for UCMA that does not involve persuasiveness (subject to the above taboos) wants to chime in I’d love to hear it. Otherwise, I think the thread has reached its (happy) conclusion.
If anyone who does think they have a coherent definition for UCMA that does not involve persuasiveness (subject to the above taboos) wants to chime in I’d love to hear it.
I’ll give it a shot: an argument is universally compelling if no mind both a) has reasons to reject it, and b) has coherent beliefs. This is to say that a mind can only believe that the argument is false by believing a contradiction.
I’ll give it a shot: an argument is universally compelling if no mind both a) has reasons to reject it, and b) has coherent beliefs. This is to say that a mind can only believe that the argument is false by believing a contradiction.
I think this may sound stronger than it actually is, for the same reasons that you can’t convince an arbitrary mind who does not accept modus ponens that it is true.
More to the point, recall that one rationalist’s modus tollens is another’s modus ponens. This definition is defeated by any mind who possesses a strong prior that the given UCMA is false, and is willing to accept any and all consequences of that fact as true (even if doing so contradicts mathematical logic, Occam’s Razor, Bayes, or anything else we take for granted). This prior is a reason to reject the argument (every decision to accept or reject a conclusion can be reduced to a choice of priors), and since it is willing to abandon all beliefs which contradict its rejection it will not hold any contradictory beliefs. It’s worth noting that “contradiction” is a notion from formal logic which not all minds need to hold as true; this definition technically imposes a very strong restriction on the space of all minds which have to be persuaded. The law of non-contradiction (~(A ^ ~A) ) is a UCMA by definition under that requirement, even though I don’t hold that belief withcertainty.
The arbitrary choice of priors, even for rational minds, actually appears to defeat any UCMA definition that does not beg the question. Of course, it is also true that any coherent definition begs the question one way or another (by defining which minds have to be persuaded such that it either demands certain arguments be accepted by all, or such that it does not). Now that I think about it, that’s the whole problem with the notion from the start. You have to define which minds have to be persuaded somewhere between a tape recorder shouting “2 + 2 = 5!” for eternity and including only your brain’s algorithm. And where you draw that line determines exactly which arguments, if any, are UCMAs.
And if you don’t have to persuade any minds, then I hesitate to permit you to call your argument “universally compelling” in any context where I can act to prevent it.
the argument “A implies B” is not universally compelling unless every rational agent must accept that “P(B | A) > P(B | !A)”
More colloquially, one property of universally compelling evidence might be that all rational agents must agree on the particular direction a particular piece of evidence should adjust a particular prior.
You’re just passing the recursive buck over to “rational”. Taboo rational, and see what you get out; I suspect it will be something along the lines of “minds that determine the right direction to shift the evidence in every case”, which, notably, doesn’t include humans even if you assume that there is an objectively decidable “rational” direction. There is no objectively determinable method to determine what the correct direction to shift is in any case; imagine an agent with anti-occamian priors, who believes that because the coin has come up heads 100 times in a row, it must be more likely to come up tails next time. It’s all a question of priors.
I think there is an objectively right direction to shift, given particular priors. Your anti-regularity observer seems to be making a mistake by becoming more confident if he actually sees heads come up next.
Also, I edited my post above to fix a notational error.
This prior is a reason to reject the argument (every decision to accept or reject a conclusion can be reduced to a choice of priors), and since it is willing to abandon all beliefs which contradict its rejection it will not hold any contradictory beliefs.
You’re right that I am committed to denying this, though I would also point out that it does not follow a priori that it is always possible to resolve the state of having contradictory beliefs by rejecting either side of a contradiction arbitrarily. However, in order to deny the above, I must claim that there are some beliefs a mind holds (or is committed to, where this means that these beliefs are deductively provable from what the mind does believe) just in virtue of being a mind. I’ll bite that bullet, and claim that there exists a UCMA of this kind. I also think the Law of Non-Contradiction is a UCA, and in fact it’s trivially so on my definition, but I think that’ll hold up: there are no Bayesian reasons to think that ascribing it a probability of 1 is a problem, and I do think I can defend the claim that evidence against it is a priori impossible (EY’s example reasons for doubt in the two articles you cite wouldn’t apply in this case).
You have to define which minds have to be persuaded somewhere between a tape recorder shouting “2 + 2 = 5!” for eternity and including only your brain’s algorithm. And where you draw that line determines exactly which arguments, if any, are UCMAs.
This isn’t a problem on my definition of a UCA. My understanding of a UCA (which I think represents an honest to god position, namely Kant’s) is consistant with any given mind believing the UCA to be false, perhaps because of reasons like the tape-recorder. Only, such a mind couldn’t have consistant beliefs.
And if you don’t have to persuade any minds, then I hesitate to permit you to call your argument “universally compelling” in any context where I can act to prevent it.
Remember that my definition of a UCMA isn’t ‘any mind under any circumstances could always be persuaded’. To attack this view of UCMAs is, I think, to attack a strawman. If we must take UCMAs to be arguments which are universally and actually persuasive for any mind in any circumstance in order to see EY’s point (here or elsewhere) as valid, then this is a serious critique of EY.
Be very, very cautious assigning probability 1 to the proposition that you even understand what the Law of Contradiction means. How confident are you that logic works like you think it works; that you’re not just spouting gibberish even though it seems from the inside to make sense. If you’d just had a major concussion, with severe but temporary brain damage, would you notice? Are you sure? After such damage you might claim that “if bananas then clocks” was true with certainty 1, and feel from the inside like you we’re making sense. Don’t just dismiss minds you can’t empathize with (meaning minds which you can’t model by tweaking simple parameters of your self-model) as not having subjective experiences that look, to them, exactly like yours do to you. You already know you’re running on corrupted hardware; you can’t be perfectly confident that it’s not malfunctioning, and if you don’t know that then you can’t assign probability 1 to anything (on pain of being unable to update later).
Again, though, you’ve defined the subspace of minds which have to be persuaded in a way which defines precisely which statements are UCAs. If you can draw useful inferences on that set of statements then go for it, but I don’t think you can. Particularly worth noting is that there’s no way any “should” statement can be a UCA because I can have any preferences I want and still fit the definition, but “should” statements always engage with preferences.
How confident are you that logic works like you think it works; that you’re not just spouting gibberish even though it seems from the inside to make sense.
I’m not even 90% sure of that, but I am entirely certain that the LNC is true: suppose I were to come across evidence to the effect that the LNC is false. But in the case where the LNC is false, the evidence against it is also evidence for it. In fact, if the LNC is false, the LNC is provable, since anything is provable from a contradiction. So if its true, it’s true, and if it’s false, it’s true. So it’s true. This isn’t entirely uncontroversial, there is Graham Priest after all.
Particularly worth noting is that there’s no way any “should” statement can be a UCA because I can have any preferences I want and still fit the definition, but “should” statements always engage with preferences.
I’ll channel Kant here, cause he’s the best UCMAist I know. He would say that almost all ‘should’ statements involve preferences, but not all. Most ‘should’ statements are hypothetical: If you want X, do Y. But one, he says, isn’t, it’s categorical: Do Y. But there’s nothing about ‘should’ statements which a priori requires the input of preferences. It just happens that most of them (all but one, in fact) do.
Now, Kant actually doesn’t think the UCMA is UC for every mind in mind-space, though he does think it’s UC for every mind capable of action. This is just to say that moral arguments are themselves only applicable to a subset of minds in mind-space, namely (what he calls) finite minds. But that’s a pretty acceptable qualification, since it still means the UCMA is UC for everything to which morality is relevant.
I’m not even 90% sure of that, but I am entirely certain that the LNC is true: suppose I were to come across evidence to the effect that the LNC is false. But in the case where the LNC is false, the evidence against it is also evidence for it. In fact, if the LNC is false, the LNC is provable, since anything is provable from a contradiction. So if its true, it’s true, and if it’s false, it’s true. So it’s true. This isn’t entirely uncontroversial, there is Graham Priest after all.
You say you’re not positive that you know how logic works, and then you go on to make an argument using logic for how you’re certain about one specific logical proposition. If you’re just confused and wrong, full stop, about how logic works then you can’t be sure of any specific piece of logic; you may just have an incomplete or outright flawed understanding. It’s unlikely, but not certain.
Also, you seem unduly concerned with pointing out that your arguments are not new. It’s not anti-productive, but neither is it particularly productive. Don’t take this as a criticism or argument, more of an observation that you might find relevant (or not).
The Categorical Imperative, in particular, is nonsense, in at least 2 ways. First, I don’t follow it, and have no incentive to do so. It basically says “always cooperate on the prisoner’s dilemma,” which is a terrible strategy (I want to cooperate iff my opponent will cooperate iff I cooperate). It’s hardly universally compelling since it carries neither a carrot nor a stick which could entice me to follow it. Second, an arbitrary agent need not care what other minds do. I could, easily, prefer that a) I maximize paperclips but b) all other agents maximize magnets. These are not instrumental goals; my real and salient terminal preferences are over the algorithms implemented not the outcomes (in this case). I should break the CI since what I want to do and what I want others to do are different.
Also, should statements are always descriptive, never prescriptive (as a consequence of what “should” means). You can’t propose a useful argument of the sort that says I should do x as a prescription. Rather you have to say that my preferences imply that I would prefer to do x. Should is a description of preferences. What would it even mean to say that I should do x, but that it wouldn’t make me happier or fulfill any other of my preferences, and I in fact will not do it? The word becomes entirely useless except as an invective.
I don’t really want to go into extreme detail on the issues with Kantian erhics; I’m relatively familiar with it after a friend of mine wrote a high school thesis on Kant, but it’s full of elementary mistakes. If you still think it’s got legs to stand I recommend reading some more of the sequences. Note that human morality is written nowhere except in our brains. I’m tapping out, I think.
Okay, fair enough. You’ve indulged me quite a ways with the whole UCMA thing, and we finished our discussion of EY’s sequence argument, so thanks for the discussion. I’ve spent some years studying Kant’s ethical theory though, so (largely for my own enjoyment) I’d like to address some of your criticisms of the CI in case curiosity provokes you to read on. If not, again, thanks.
I don’t really want to go into extreme detail on the issues with Kantian erhics; I’m relatively familiar with it after a friend of mine wrote a high school thesis on Kant, but it’s full of elementary mistakes.
This conclusion should set off alarm bells: if I told you I’d found a bunch of elementary mistakes in the sequences, having never read them but having discussed them with an acquaintance, you would bid me caution.
First, I don’t follow it, and have no incentive to do so.
The issue of incentive is one that Kant really struggles with, and much of his writings on ethics following the publication of the Groundwork for the Metaphysics of Morals (where the CI is introduced) is concerned with this problem. So while on the one hand, you’re correct to think that this is a problem for Kant, it’s also a problem he spent a lot of time thinking about himself. I just can’t do it any justice here, but very roughly Kant thinks that in order to rationally pursue happiness, you have to pursue happiness in such a way that you are deserving of it, and only by being morally good can you deserve happiness. This sounds very unconvincing as read, but Kant’s view on this is both sophisticated and shifting. I don’t know that he felt he ever had a great solution, and he died writing a book on the importance of our sense of aesthetics and its relation to morality.
It basically says “always cooperate on the prisoner’s dilemma,”
The CI is not a decision theory, nor is a decision theory a moral theory. It’s important not to confuse the two. If you gave Kant the prisoner’s dilemma, he would tell you to always defect, because you should always be honest. You would be annoyed, because he’s mucking around with irrelevant features of the set up, and he would point out to you that the CI is a moral theory and that the details of the setup matters. The CI says nothing consistant or interesting about the prisoner’s dilemma, nor should it.
I could, easily, prefer that a) I maximize paperclips but b) all other agents maximize magnets.
You could, and that’s how preferences work. So there could be no universal hypothetical imperative. But the categorical imperative doesn’t involve reference to preferences. If you take yourself to have a reason to X, which makes no reference to preferences (terminal or otherwise), you at the same time take any arbitrary reasoner to have a reason to X. Suppose, for comparison, that a set of minds in mind-space happened to (against whatever odds) have exactly the same evidence as you for a proposition. You couldn’t coherently believe that you had reason to believe the proposition, but that they did not. Reasons don’t differentiate between reasoners that way.
You may think imperatives always make reference to preferences, but this is an argument you’d have to have with Kant. It’s not a priori obvious or anything, so it’s not enough to state it and say ‘Kant is wrong’.
I should break the CI since what I want to do and what I want others to do are different.
The CI is not the claim that everyone should do what you want to do. The CI is the demand (essentially) that you act on reasons. The structure of reasons (like the fact that reasons don’t discriminate between reasoners) gives you the whole ‘universal’ bit.
First off, the quoted argument was, as far as I can tell, entirely meant as an illustrative abstraction. The culprit here is the devious function X().
Suppose I take the set of all possible logically coherent statements that could be made about any given mind. Within this set, ‘X’ is any given statement about one mind. X(m) represents whether this given statement is True, False or Undefined / Undecidable for this mind ‘m’.
For all X1..Xn, for a given mind ‘m1’, find all the X that are true. Then for all X() for m2, find those that are true. Supposing any given X has 50% probability of being true of any given m, then X1 being true for m1 has probability 0.5, being true for both m1 and m2 has probability 0.25, and so on dividing the odds by 2 for each additional mind for which the conjunctive must hold true.
So for any given X, for m1..m(10^12), X has (1 / 2^12) probability of being true if we assume a priori 50% chance of that statement being true.
Conversely, for any given X, X(m) will be true for at least one m with 2^12 / (2^12 + 1) probability.
The central inference in the argument is that we do not know the structure of ‘all possible minds’ or of ‘all possible arguments’, but it is reasonable to believe that the space of all possible minds is sufficiently large and versatile that the subset A() of all possible statements X(), where A() are statements of the form “A(m) is true if mind ‘m’, when presented with the argument A, will change some specific belief / internal state / thought / behavior to state Y”, is not true for all A(m). This latter part of the argument rests mostly on the following reasoning:
If there is any given argument A that will convince all currently known minds such that A(m) is true, and all known minds accept the argument, we can almost certainly construct a mind nearly identical to one of these m, but for which the input A is forbidden, or that will self-destruct immediately upon receiving it, or where the first and only possible result of A(m) is always false, or where arguments of the form which A takes are simply incoherent.
For example, if A is a UCMA of the form where you speak or write certain words and have the listener hear or read them, make the listener unable to understand the language. If the UCMA implies translation to a given language of choice that the listener understand, craft the listener so that the listener does not understand or have any languages. If the UCMA is some form of complex hacking by specifying complicated inputs that abuse internal properties of the mind in question… well, how likely is it that all possible minds share these exact internal properties?
This is a complex subject since “minds” and “arguments” are loaded terms with lots of anthropomorphic stuff hidden behind them, and much of the actual meat of the problem lies in things philosophers still disagree on despite overwhelming probability.
Excellently explained, thank you. The argument you present seems to me to be on the whole reasonable, but it involves two assumptions no UCMA enthusiast I know of would ever accept.
Supposing any given X has 50% probability of being true of any given m...
And
“A(m) is true if mind ‘m’, when presented with the argument A, will change some specific belief / internal state / thought / behavior to state Y”
These two assumptions aren’t argued for, nor are they attributed to any UCMA enthusiast, so I cant see any reason why she should accept them. Do they seem plausible to you? If so, can you give me reasons to accept them?
Supposing any given X has 50% probability of being true of any given m (...)
This isn’t a direct claim of fact, but a flat assumption to simplify illustration and calculations. The same argument extends for any arbitrary probability by showing that mathematically, no matter the probability in question (as long as it is a probability, and not a 0 or 1), as the number of possible minds grows towards infinity, the chance of X being true for all minds keeps decreasing in a similar manner.
The hidden assumption behind this, of course, is that I have a high prior that the number of different possible minds is sufficiently high and the probability comparably low enough for this compound probabilistic growth to become critical. Since the number of known different minds already exceeds seven billion and any given random statement of the form “Mind ‘m’ believes that X is immoral under Y circumstances or context Z” is extremely unlikely to be true for any given mind (and like above, scales to nigh-infinitesimal when conjugating it across all seven billion), I think this hidden assumption is a very reasonable one. An example for clarity:
Mind M ( John B Gato ) believes ( Looking at Jello ) is immoral in context ( Five days and seven minutes after every new moon for a period of three hours, or while scratching one’s toe. )
Of course, this is an intuition about moral beliefs, not about being-convinced-by-arguments, but it’s an intuition about the diversity of ways human minds process arguments that hints at the possible diversity among non-human mind structures.
“A(m) is true if mind ‘m’, when presented with the argument A, will change some specific belief / internal state / thought / behavior to state Y”
This is my own model / abstraction of Argument—Mind—Belief/Action. If a UCMA supporter does not believe that arguments lead to any change of belief or behavior in a mind once the argument is made to that mind, then that seems to directly contradict the very idea of a universally compelling argument that persuades any mind.
So the quote above is a model for “If the mind is compelled by the argument, it will have a certain property which allows the argument to compell it” (this property may be a simple emergent property of all possible minds that arises from a combination of the basic necessary properties of all minds, or might be implicit in the structure of logic, which I reckon is a main argument for some of the more sophisticated UCMA supporters).
It seems very unlikely to me that a UCMA enthusiast would grant that a UCMA has in any given case only a fifty percent chance of being UC. So to assume this begs the question against them. It may be that the UCMAist is being silly here, or that the burden is on them to show that things are otherwise, but that’s not relevant to the question of the strength of EY’s argument against UCMAs.
...I think this hidden assumption is a very reasonable one.
It is, but it’s a bit too reasonable: that is, it’s unreasonable to think that the UCMAist actually thinks that the UCMA is already explicitly accepted by everyone, or even that everyone could be immediately or in any circumstances persuaded that the UCMA is true. UCMA’s on this conception are obviously false, but then EY’s argument is wholly trivial. Nor would we need an argument: it is not hard to come up with a single case of moral disagreement, and that’s all that would be necessary. But this would be to attack a stawman.
The UCMAist is committed to some sense being given to the UC bit, you’re right. If we go to an actual UCMAist, like Kant, the explanation looks something like this: People say all sorts of things about their moral beliefs, but no one could have reasons to doubt the UCMA while holding consistant beliefs. This means that in principle, any mind could be persuaded to accept the UCMA, but not any mind under any circumstances. I (Kant) am committed to saying that every mind is so structured that the UCMA is an unavoidable deductive conclusion, not that every mind in every circumstance has or would arrive at the UCMA. So if this is what being a UCMA means:
“A(m) is true if mind ‘m’, when presented with the argument A, will change some specific belief / internal state / thought / behavior to state Y”
Then yes, UCMA’s are impossible. But no one has ever thought otherwise, and it remains open whether something very much like them, namely moral arguments which every possible mind is committed to accepting (whether or not they do accept it) is possible.
It seems very unlikely to me that a UCMA enthusiast would grant that a UCMA has in any given case only a fifty percent chance of being UC. So to assume this begs the question against them. It may be that the UCMAist is being silly here, or that the burden is on them to show that things are otherwise, but that’s not relevant to the question of the strength of EY’s argument against UCMAs.
No no no. The point of the argument is that it doesn’t matter what the probability is. Even if it’s not 50%, the dynamics at work still make us end up with and exponentially small probability that something is universally compelling, just with the raw math.
The burden is on the UCMAist to show that there are structural reasons why minds must necessarily have certain properties that also happen to coincide with the ability to received, understand, and be convinced by arguments, and also coincide with the specific pattern where at least one specific argument will result in the same understanding and the same resulting conviction for all possible minds.
Both of these are a priori extremely unlikely due to certain intuitions about physics and algorithms and due to the mathematical argument Eliezer makes, respectively.
namely moral arguments which every possible mind is committed to accepting (whether or not they do accept it) is possible.
I’d require clarification on what is meant by “committed to accepting” here. They accept the argument and change their beliefs, or they do not accept the argument and do not change their beliefs. For either case, they either do this in all situations or only some situations. They may sometimes accept it and sometimes not accept it.
The Kant formulation you give seems explicitly about humans, only humans and exclusively humans and nothing else. The whole point of EY’s argument against UCMAs is that there are no universally compelling arguments you could make to an AI built in a manner completely alien to humans that would convince the AI that it is wrong to burn your baby and use its carbon atoms to build more paperclips, even if the AI is fully sentient and capable of producing art and writing philosophy papers about consciousness and universally-compelling moral arguments.
There’s other things I’d say are just wrong about the way this description models minds, but I think that for now I’ll stop here until I’ve read some actual Kant or something.
The point of the argument is that it doesn’t matter what the probability is.
Right, but I can’t imagine a UMCAist thinking this is a matter of probability. That is, the UMCAist will insist that this is a necessary feature of minds. The burden may be up to them, but that’s not EY’s argument (its not an argument against UMCA’s at all). And I took EY to be giving an argument to the effect that UMCA’s are false or at least unlikely. You may be right that EY has successfully argued that if one has no good reasons to believe a UMCA exists, the probability of one existing must be assessed as low. But this isn’t a premise the UMCAist will grant, so I don’t know what work that point could do.
The Kant formulation you give seems explicitly about humans, only humans and exclusively humans and nothing else.
You might be able to argue that, bu that’s not the way Kant sees it. Kant is explicit that this applies to all minds in mind-space (he kind of discovered the idea of mind-space, I think). As to what ‘committed to accepting’ means, you’re right that this needs a lot of working out, working out I haven’t done. Roughly, I mean that one could not have reasons for denying the UMCA while having consistant beliefs. Kant has to argue that it is structural to all possible minds to be unable to entertain an explicit contradiction, but that’s at least a relatively plausible generalization. Still, tall order.
On the whole, I entirely agree with you that a) the burden is on the UCMAist, b) this burden has not been satisfied here or maybe anywhere. I just wanted to raise a concern about EY’s argument in this post, to the effect that it either begs the question against the UCMAist, or that it is invalid (depending on how it’s interpreted). The shortcomings of the UCMAist aren’t strictly relevant to the (alleged) shortcomings of EY’s anti-UCMAist argument.
There’s an argument in the metaethics sequence, to the effect that there are no universally compelling moral arguments. This argument seems to be an important cashed thought (in don’t mean that in any pejorative sense) in LW discussions of morality. This argument also seems to me to be faulty. Can anyone help me see what I’m missing?
The argument is from No Universally Compelling Arguments:
The central inference in the argument seems to me to go like this:
P1) Any universal generalization over minds (‘All minds m: X(m)’) is very unlikely to be true.
P2) A purportedly universally compelling moral argument has the form ‘All minds m: X(m)’
C) A purportedly universally compelling moral argument is very unlikely to be true.
The reason I think this is faulty is that P1 is itself an argument of the form ‘All minds m: X(m)’, that is, it’s a universal generalization over minds. If that’s so, then P1 is very unlikely to be true, and we shouldn’t accept the argument. In order to save the argument, we would have to weaken P1 to cover a more specific set of generalizations over minds (so that P1 itself is excluded) but if we do this, then the argument is invalid, since universally compelling moral arguments may end up excluded as well. We might have good reasons for thinking they won’t be, but no such reasons are given in the sequence post.
I don’t see how your P1 is a statement over all minds, it looks more like a statement over most arguments.
Agreed. P1 is quantifying over arguments, not over minds.
I see the symmetry between P1 and a universally compelling moral argument in this: they both make a claim about the application of an argument quantifying over all minds in mind-space.
The claim EY is refuting is ‘For all minds m, m: (moral argument X is compelling)m.’
P1 makes the claim ‘For all minds m, m:(an argument of the form ‘for all minds m:X(m) is unlikely to be true)m.’
Is that not right?
It looks like your P1 is quantifying twice over the same variable. I don’t think that’s right.
Is it? I intended it to only quantify over the non-nested m. Am I committed to quantifying over the nested m as well?
Now I’m just confused by your syntax.
Or, more likely, I am confused by my syntax. If you were to formalize EY’s argument, how would you put it?
At the risk of prolonging an unproductive thread, I’d say P1 is like
P1: For most predicates X: Not (For all minds m: X(m))
This isn’t self-refuting.
Thanks, you’re right that this isn’t self refuting. But with that P1, the argument seems invalid:
P1: For most predicates X: Not (For all minds m: X(m))
P2: UCMAs are X
C: Not UMCA
is like
P1: For most prime numbers n: (odd)n
P2: 2 is prime
C: 2 is odd
Edit: you might think that the conclusion is not that not ‘not UMCA’ but ‘UMCA is unlikely’, but this doesn’t follow either. I don’t know quite how ‘most’ quantifiers work, but I don’t think we can read a probabilistic conclusion off of them. I don’t think it follows from the above, for example, that 2 is likely to be odd.
Yes, the crucial issue in this conversation is the concept of ‘most’ and ‘probability’. What you can conclude from P1 is that a priori, a randomly selected predicate X probably does not satisfy X(m) for all m. If we had other reasons to believe that X(m) for all m, then we can update our beliefs. Similarly, we expect that a randomly selected prime number n is probably odd; but if we learn the further fact that n=2, then our belief changes.
So what do you make of this argument then? Suppose I were of the opinion that 2 is an even prime. You come to me with an argument to the effect that I should not believe 2 to be prime because a randomly selected prime number is very, very unlikely to be even. Should I be convinced by that? I may be convinced that in some sense, 2 is unlikely to be even, but I don’t think I should accept that 2 is not even, or that the evenness of 2 is questionable.
Similarly, suppose someone believes an argument to be universally compelling. It seems to me that EY’s argument should be unmoving: granting that it is unlikely for a randomly selected argument to be UC, but theirs is no randomly selected argument. And on DaFranker’s reading of this argument, the thesis that a given X is unlikely to hold for of all minds relies on the assumption that for most X’s, there is (something like) a 50% chance of its being true of some mind. But certainly a UCMAist won’t accept that this is true of UCMA’s. UCMA’s, they will say, are exactly those X’s for which this is not true.
The burden may be on them to justify the possibility of such an X, but that fact won’t save the argument.
As for your first paragraph, well, this is a straightforward application of Bayes’ theorem. If you’re sure that 2 is even, then learning that 2 was randomly selected from some distribution over primes should not be enough to change your credence very much.
As for your second and third paragraphs: Yes, the argument of Eliezer you’re talking about doesn’t refute the existence of universally compelling arguments; it merely means that you shouldn’t believe you have a universally compelling argument unless you have a good reason for believing so. If you think you have a good reason, then you don’t have to worry about this argument.
There’s a very simple argument refuting the existence of universally compelling arguments, and I believe it was stated elsewhere in this thread. It’s that argument you have to refute, not this one.
Please point this out to me if you get a chance, as I haven’t noticed it. And thanks for the discussion. I mean that: I can see that this wasn’t helpful or interesting for you, but rest assured it was for me, so your indulgence is appreciated.
You’re welcome! The refutation of universally compelling arguments I was referring to is this one. I see you responded that you’re interested in a different definition of “compelling”. On the word “compelling”, you say
This is indeed the meaning of “compelling” that Eliezer uses, and Eliezer’s original argument is indeed trivial, which perhaps explains why he spent so few words on it.
If you wanted to defend a different claim, that there are arguments that all minds are “rationally committed” to accepting or whatever, then you’d have to begin by operationalizing “committed”, “reasons”, etc. I believe there’s no nontrivial way to do this. In any case the burden is on others to operationalize these concepts in an interesting way.
Okay, thanks for pointing that out.
Why would you want to formalize the argument?
That I can’t argue with, though it wouldn’t follow from that that UCMAs are likely to be false.
EDIT: you edited your post, and so my reply doesn’t seem to make sense. In answer to your new question, I would say ‘I don’t, I just want some presentation of the argument on which its validity (or invalidity) is obvious’.
UCMA is making a claim about all minds, P1 is making a claim about some undefined subset of all minds.
They both talk about “all minds,” but only one of them makes a claim -about- all minds.
A parallel pair of arguments might be: All squares are rectangles The claim that all squares are rectangles is unlikely to be true of all squares.
The first claim is stronger than the second, and requires more proof. The fact that we can in fact prove it is irrelevant, and part of why I chose this example; consider the inverse propositions that all rectangles are squares, and that that claim is unlikely to be true, to see why this is important.
This is analogous to the conclusion of the above argument, not P1. An analogue to P1 would have to be something like ‘Any argument of the form ‘for all squares s:(X)s is unlikely to be true.’ The question would then be this: does this analogue of P1 count as an argument of the form ‘s:(X)s’? That is, does it quantify over all squares?
You might think it doesn’t, since it just talks about arguments. But my point isn’t quite that it must count as such an argument, but rather that it must count as an argument of the same form as P2 (whatever that might be). The reason is that P2 is not like ‘all squares are rectangles’. If it were, P2 would be a (purportedly) universally compelling moral argument. But P2 is rather the claim that there is such an argument. P2 is ‘for all minds m:(Moral Argument X is compelling)m’.
I see what you’re talking about. My confusion originates in your definition of P2, rather than P1, where I thought the confusion was originated.
Suppose two minds, A and B. A has some function for determining truth, let’s call it T. Mind B, on the other hand, is running an emulation of mind A, and its truth function is not(T).
Okay, yes, this is an utterly pedantic kind of argument, but I think it demonstrates that in -all- of mindspace, it’s impossible to have any universally compelling argument, without relying on balancing two infinities (number of possible arguments and number of possible minds) against each other and declaring a winner.
That sounds pretty good to me, though I think it’s an open question whether or not what you’re talking about is possible. That is, a UCMA theorist would accuse you of begging the question if you assumed at the outset that the above is a possibility.
It’s only an open question insofar as what are considered “minds” and “arguments” remain shrouded in mystery.
I’m rather certain that for a non-negligible fraction of all minds, the entire concept of “arguments” is nonsensical. There is, after all, no possible combination of inputs (or “arguments”), that will make the function “SomeMind(): Print 3” output that it is immoral to tube-feed chicken.
Why are you certain of this?
Because of my experience with programming and working with computation, I find it extremely unlikely that, out of all possible things, the specific way humans conceptualize persuasion and arguments would be a necessary requirement for any “mind” (which I take here as a ‘sentient’ algorithm in the largest sense) to function.
If the way we process these things called “arguments” is not a requirement for a mind, then there almost certainly exists at least one logically-possible mind design which does not have this way of processing things we call “arguments”.
As another intuition, if we adopt the Occam/Solomonoff philosophy for what is required to have a “mind”, then something as complicated as the process of understanding arguments, being affected, influenced or persuaded by them, by running through filters and comparing with prior knowledge and so on until some arguments convince or do not convince… that must be required for all possible minds as a component of an already-complex system called “minds”… sounds extremely much less common in the realm of all possible universes than the universes where simpler minds exist that do not have this property of understanding arguments and being moved by them.
I don’t have any experience with programming at all, and that may be the problem: I just don’t see these reasons. To my mind (ha) a mind incapable of processing arguments, which is to say holding reasons in rational relations to each other or connecting premises and conclusions up in justificatory relations or whatever, isn’t reasonably called a mind. This may just be a failure of imagination on my part So...
Could you explain this? I’m under the impression that being capable of solomonoff induction requires being capable of 1) holding beliefs, 2) making inferences about those beliefs, 3) changing beliefs. Yet this seems to me to be all that is required for ‘understanding and being convinced by an argument’.
In my limited experience, UCMA supporters explicitly rejected the assertion that “arguments” and “being convinced by an argument” are equivalent to “evidence” and “performing a bayesian update on evidence”. So those three would be enough for evidence and updates, but not enough for argument and persuasion according to my next-best-guess of what they mean by “argument” and “convinced”.
For one, you need some kind of input system, and some process that looks at this input and connects it to pieces of an internal model, which requires and internal model and some structure that sends signals from the input to the process, and some structure where the process has modification access to other parts of the mind (to form the connections and perform the edits) in some way.
Then you need something that represents beliefs, and some weighing or filtering system where the elements of the input are judged (compared to other nodes in the current beliefs) and then evaluated using a bunch of built-in or learned rules (which implies having some rules of logic built-in to the structure of the mind, or the ability to learn such rules, both of which are non-trivial complexity-wise), and then those evaluations organized in a way where it can be concluded whether the argument is sound or not, and the previous judgments of the elements integrated so that it can be concluded whether the premises are also good, and then the mind also requires this result to send a signal to some dynamic process in the brain that modus ponens the whole thing into using the links to the concepts and beliefs to update and edit them to the new values prescribed by the compelling argument.
Whew, that’s a lot of stuff that we need to design into our mind that seems completely unnecessary for a mind to have sentience, as far as I can tell. I sure hope we don’t live in the kind of weird universes where sentience necessarily implies or requires all of the above!
Which is where the Occam/SI comes in. All of the above is weird, very specific, and extremely complex in most machine designs I can think of. Sentience is itself complex, but doesn’t seem to require the above as far as we can tell. Positing that minds also require all these additional complexities seems like a very bad idea. Statistically, ‘A’ is always more likely than ‘A and B’. Positing UCMA is a bit akin to positing ‘A and B and C and Fk but not Re and not any of Ke through Kz and L1..273 except L22’.
Eh, for the UCMA arguments I’m familiar with, they would be happy to work within the (excellent) Solomonoff framework as long as you allowed for probabilities of 0 and 1. I get that this isn’t an unproblematic allowance, but nothing about the math actually requires us to exclude probabilities of 0 and 1 (so far as I understand it).
What is necessary? It’ll pay off for us to get this on the table.
If we knew exactly, someone would have a nobel for it and the nonperson predicate would be a solved problem by now, along with the Hard Problem of Consciousness and a throng of other things currently puzzling scientists the world over.
However, we do have a general idea of the direction to take, with an example here of some of the things involved. There’s still the whole debate and questions around the so-called “hard problem of consciousness”, but overall it doesn’t even seem as if the ability to communicate is required for consciousness or sentience, let alone hold the ability to parse language in a form remotely close to ours or that allows anything akin to an argument as humans are used to the word.
But past that point, the argument is no longer about UCMAs, and becomes about morality engines (and whether morality or something akin to it must exist in all minds), consciousness, what constitutes an ‘argument’ and ‘being convinced’, and other things humans yet understand so very little about.
Okay, I see the problem. Let’s say this: within the whole of mind-space there is a subset of minds capable of morally-evaluable behavior. For all such minds, the UCMA is true. This may be a tiny fraction, but the UCMAist won’t be disturbed by that: no UCMAist would insist that the UCMA is UC for minds incapable of anything relevant to morality. How does that sound?
This sounds like a good way to avoid the heavyweight problems with all the consciousness debates, so it seems like a good idea.
However, it retains the problem of defining “morality”, which is still unresolved. UCMAists will argue from theories of morality where UC is an element of the theory, while E.Y. already assumes a different metaethics where there is no clear boundaries of human “morality” and where morality-in-the-way-we-understand-it is a feature of humans exclusively, and other things might have things akin to morality that are not morality, and some minds would be able to evaluate moral behaviors without caring about morality in the slightest, while some other minds we might consider morally-important and yet would completely ignore any “UCMA” that would otherwise compel any human.
Without going into the details, you could hypothesize a simple mind than automatically rejects any argument. This would by itself prove the No Universally Compelling Arguments theory.
That would do it, though it may only attack a straw man: the thesis that the, say, categorical imperative is universally compelling is not the thesis that the CI is universally persuasive. Rather, I think the thought is that we are all rationally committed to the CI, whether we know or admit this or not.
Taboo compelling and restate. If compelling does not mean persuasive then what does it mean to you? Also taboo “committed” and “rational”—I think there’s a namespace conflict over your use of rational and the common Less Wrong usage, so restate using different terms. As a hint, try and imagine what a universally compelling argument would look like. What properties does it have? How do different minds react to understanding it, assuming they are capable of doing so? For bonus points explain what it means to be rationally committed to something (without using those words or synonyms).
Also worth noting: P1 is a generalization over statements about minds, not minds.
Well, we have two options in tabooing ‘compelling’. On the one hand, we could mean ‘persuasive’ where this means something like ‘If I sat down with someone, and presented the moral argument to them, they would end up accepting it regardless of their starting view’. This seems to be a bad option, because the claim that ‘there are no universally persuasive moral arguments’ is trivial. No one (of significance) has ever held the contrary view.
So our other option is to take ‘compelling’ as something like what Kantians say about the CI, namely that every mind is committed to it, whether they accept this or not (‘not’ out of irrationality). As you say, this leaves us with a lot more tabooing and explaining to do. I’m happy to go on with this, since it’s the sort of thing I enjoy, but it is a digression from my (perhaps confused) complaint about EY’s argument. The important bit there is just that ‘compelling’ probably shouldn’t be taken in such a way as to make EY’s point trivial.
.
The problem here is that the second option you offer does nothing to explain what a compelling argument is; it just passes the recursive buck onto the word “committed”. I know you said you recognize that, but unless we can show that this line of reasoning is coherent (let alone leads to a relevant conclusion, let alone correct) then there’s no reason to assume that Eliezer’s point isn’t trivial in the end. Philosophers have believed a lot of silly things, after all. The only sensible resolution I can come up with is where you take “committed to x” to mean “would, on reflection and given sufficient (accurate) information and a great deal more intelligence, believe x”. The problem is that this is still trivially false in the entirety of mindspace. You might, although I doubt it, be able to establish a statement of that form over all humans (I think Eliezer disagrees with me on the likelihood here). You could certainly not establish one given a mindspace that includes both humans and paper clip maximizers.
If what you’re saying is this, then we agree: EY doesn’t here present an argument that UCMAs are likely to be false, but he does successfully argue that a certain class of generalizations over mind-space are likely to be false (such as generalizations about what minds will find persuasive) along with the assumption that a UCMA will fall into that class.
If that’s the line, then I think the argument is sound so far as it goes. UCMA enthusiasts (I am not among them, but I know them well) will not accept the final assumption, but you may be right that the burden is on them to show that UCMA’s (whatever ‘compelling’ is supposed to mean) does not fall into this class.
Alternatively, we could just posit that we’re only arguing against those people who do accept the assumption, that is those people who do take ‘compelling’ in UCMA to mean something like ‘immediately persuasive’, but then we’re probably tilting at windmills.
I suspect that our beliefs are close enough to each other at this point that any perceived differences are as likely to be due to minor linguistic quibbles as to actual disagreement. Which is to say, I wouldn’t have phrased it like you did (had I said it with that phrasing I would disagree) but I think that our maps are closer than our wording would suggest.
If anyone who does think they have a coherent definition for UCMA that does not involve persuasiveness (subject to the above taboos) wants to chime in I’d love to hear it. Otherwise, I think the thread has reached its (happy) conclusion.
I’ll give it a shot: an argument is universally compelling if no mind both a) has reasons to reject it, and b) has coherent beliefs. This is to say that a mind can only believe that the argument is false by believing a contradiction.
I think this may sound stronger than it actually is, for the same reasons that you can’t convince an arbitrary mind who does not accept modus ponens that it is true.
More to the point, recall that one rationalist’s modus tollens is another’s modus ponens. This definition is defeated by any mind who possesses a strong prior that the given UCMA is false, and is willing to accept any and all consequences of that fact as true (even if doing so contradicts mathematical logic, Occam’s Razor, Bayes, or anything else we take for granted). This prior is a reason to reject the argument (every decision to accept or reject a conclusion can be reduced to a choice of priors), and since it is willing to abandon all beliefs which contradict its rejection it will not hold any contradictory beliefs. It’s worth noting that “contradiction” is a notion from formal logic which not all minds need to hold as true; this definition technically imposes a very strong restriction on the space of all minds which have to be persuaded. The law of non-contradiction (~(A ^ ~A) ) is a UCMA by definition under that requirement, even though I don’t hold that belief with certainty.
The arbitrary choice of priors, even for rational minds, actually appears to defeat any UCMA definition that does not beg the question. Of course, it is also true that any coherent definition begs the question one way or another (by defining which minds have to be persuaded such that it either demands certain arguments be accepted by all, or such that it does not). Now that I think about it, that’s the whole problem with the notion from the start. You have to define which minds have to be persuaded somewhere between a tape recorder shouting “2 + 2 = 5!” for eternity and including only your brain’s algorithm. And where you draw that line determines exactly which arguments, if any, are UCMAs.
And if you don’t have to persuade any minds, then I hesitate to permit you to call your argument “universally compelling” in any context where I can act to prevent it.
Might we say something like:
More colloquially, one property of universally compelling evidence might be that all rational agents must agree on the particular direction a particular piece of evidence should adjust a particular prior.
You’re just passing the recursive buck over to “rational”. Taboo rational, and see what you get out; I suspect it will be something along the lines of “minds that determine the right direction to shift the evidence in every case”, which, notably, doesn’t include humans even if you assume that there is an objectively decidable “rational” direction. There is no objectively determinable method to determine what the correct direction to shift is in any case; imagine an agent with anti-occamian priors, who believes that because the coin has come up heads 100 times in a row, it must be more likely to come up tails next time. It’s all a question of priors.
I think there is an objectively right direction to shift, given particular priors. Your anti-regularity observer seems to be making a mistake by becoming more confident if he actually sees heads come up next.
Also, I edited my post above to fix a notational error.
You’re right that I am committed to denying this, though I would also point out that it does not follow a priori that it is always possible to resolve the state of having contradictory beliefs by rejecting either side of a contradiction arbitrarily. However, in order to deny the above, I must claim that there are some beliefs a mind holds (or is committed to, where this means that these beliefs are deductively provable from what the mind does believe) just in virtue of being a mind. I’ll bite that bullet, and claim that there exists a UCMA of this kind. I also think the Law of Non-Contradiction is a UCA, and in fact it’s trivially so on my definition, but I think that’ll hold up: there are no Bayesian reasons to think that ascribing it a probability of 1 is a problem, and I do think I can defend the claim that evidence against it is a priori impossible (EY’s example reasons for doubt in the two articles you cite wouldn’t apply in this case).
This isn’t a problem on my definition of a UCA. My understanding of a UCA (which I think represents an honest to god position, namely Kant’s) is consistant with any given mind believing the UCA to be false, perhaps because of reasons like the tape-recorder. Only, such a mind couldn’t have consistant beliefs.
Remember that my definition of a UCMA isn’t ‘any mind under any circumstances could always be persuaded’. To attack this view of UCMAs is, I think, to attack a strawman. If we must take UCMAs to be arguments which are universally and actually persuasive for any mind in any circumstance in order to see EY’s point (here or elsewhere) as valid, then this is a serious critique of EY.
Be very, very cautious assigning probability 1 to the proposition that you even understand what the Law of Contradiction means. How confident are you that logic works like you think it works; that you’re not just spouting gibberish even though it seems from the inside to make sense. If you’d just had a major concussion, with severe but temporary brain damage, would you notice? Are you sure? After such damage you might claim that “if bananas then clocks” was true with certainty 1, and feel from the inside like you we’re making sense. Don’t just dismiss minds you can’t empathize with (meaning minds which you can’t model by tweaking simple parameters of your self-model) as not having subjective experiences that look, to them, exactly like yours do to you. You already know you’re running on corrupted hardware; you can’t be perfectly confident that it’s not malfunctioning, and if you don’t know that then you can’t assign probability 1 to anything (on pain of being unable to update later).
Again, though, you’ve defined the subspace of minds which have to be persuaded in a way which defines precisely which statements are UCAs. If you can draw useful inferences on that set of statements then go for it, but I don’t think you can. Particularly worth noting is that there’s no way any “should” statement can be a UCA because I can have any preferences I want and still fit the definition, but “should” statements always engage with preferences.
I’m not even 90% sure of that, but I am entirely certain that the LNC is true: suppose I were to come across evidence to the effect that the LNC is false. But in the case where the LNC is false, the evidence against it is also evidence for it. In fact, if the LNC is false, the LNC is provable, since anything is provable from a contradiction. So if its true, it’s true, and if it’s false, it’s true. So it’s true. This isn’t entirely uncontroversial, there is Graham Priest after all.
I’ll channel Kant here, cause he’s the best UCMAist I know. He would say that almost all ‘should’ statements involve preferences, but not all. Most ‘should’ statements are hypothetical: If you want X, do Y. But one, he says, isn’t, it’s categorical: Do Y. But there’s nothing about ‘should’ statements which a priori requires the input of preferences. It just happens that most of them (all but one, in fact) do.
Now, Kant actually doesn’t think the UCMA is UC for every mind in mind-space, though he does think it’s UC for every mind capable of action. This is just to say that moral arguments are themselves only applicable to a subset of minds in mind-space, namely (what he calls) finite minds. But that’s a pretty acceptable qualification, since it still means the UCMA is UC for everything to which morality is relevant.
You say you’re not positive that you know how logic works, and then you go on to make an argument using logic for how you’re certain about one specific logical proposition. If you’re just confused and wrong, full stop, about how logic works then you can’t be sure of any specific piece of logic; you may just have an incomplete or outright flawed understanding. It’s unlikely, but not certain.
Also, you seem unduly concerned with pointing out that your arguments are not new. It’s not anti-productive, but neither is it particularly productive. Don’t take this as a criticism or argument, more of an observation that you might find relevant (or not).
The Categorical Imperative, in particular, is nonsense, in at least 2 ways. First, I don’t follow it, and have no incentive to do so. It basically says “always cooperate on the prisoner’s dilemma,” which is a terrible strategy (I want to cooperate iff my opponent will cooperate iff I cooperate). It’s hardly universally compelling since it carries neither a carrot nor a stick which could entice me to follow it. Second, an arbitrary agent need not care what other minds do. I could, easily, prefer that a) I maximize paperclips but b) all other agents maximize magnets. These are not instrumental goals; my real and salient terminal preferences are over the algorithms implemented not the outcomes (in this case). I should break the CI since what I want to do and what I want others to do are different.
Also, should statements are always descriptive, never prescriptive (as a consequence of what “should” means). You can’t propose a useful argument of the sort that says I should do x as a prescription. Rather you have to say that my preferences imply that I would prefer to do x. Should is a description of preferences. What would it even mean to say that I should do x, but that it wouldn’t make me happier or fulfill any other of my preferences, and I in fact will not do it? The word becomes entirely useless except as an invective.
I don’t really want to go into extreme detail on the issues with Kantian erhics; I’m relatively familiar with it after a friend of mine wrote a high school thesis on Kant, but it’s full of elementary mistakes. If you still think it’s got legs to stand I recommend reading some more of the sequences. Note that human morality is written nowhere except in our brains. I’m tapping out, I think.
Okay, fair enough. You’ve indulged me quite a ways with the whole UCMA thing, and we finished our discussion of EY’s sequence argument, so thanks for the discussion. I’ve spent some years studying Kant’s ethical theory though, so (largely for my own enjoyment) I’d like to address some of your criticisms of the CI in case curiosity provokes you to read on. If not, again, thanks.
This conclusion should set off alarm bells: if I told you I’d found a bunch of elementary mistakes in the sequences, having never read them but having discussed them with an acquaintance, you would bid me caution.
The issue of incentive is one that Kant really struggles with, and much of his writings on ethics following the publication of the Groundwork for the Metaphysics of Morals (where the CI is introduced) is concerned with this problem. So while on the one hand, you’re correct to think that this is a problem for Kant, it’s also a problem he spent a lot of time thinking about himself. I just can’t do it any justice here, but very roughly Kant thinks that in order to rationally pursue happiness, you have to pursue happiness in such a way that you are deserving of it, and only by being morally good can you deserve happiness. This sounds very unconvincing as read, but Kant’s view on this is both sophisticated and shifting. I don’t know that he felt he ever had a great solution, and he died writing a book on the importance of our sense of aesthetics and its relation to morality.
The CI is not a decision theory, nor is a decision theory a moral theory. It’s important not to confuse the two. If you gave Kant the prisoner’s dilemma, he would tell you to always defect, because you should always be honest. You would be annoyed, because he’s mucking around with irrelevant features of the set up, and he would point out to you that the CI is a moral theory and that the details of the setup matters. The CI says nothing consistant or interesting about the prisoner’s dilemma, nor should it.
You could, and that’s how preferences work. So there could be no universal hypothetical imperative. But the categorical imperative doesn’t involve reference to preferences. If you take yourself to have a reason to X, which makes no reference to preferences (terminal or otherwise), you at the same time take any arbitrary reasoner to have a reason to X. Suppose, for comparison, that a set of minds in mind-space happened to (against whatever odds) have exactly the same evidence as you for a proposition. You couldn’t coherently believe that you had reason to believe the proposition, but that they did not. Reasons don’t differentiate between reasoners that way.
You may think imperatives always make reference to preferences, but this is an argument you’d have to have with Kant. It’s not a priori obvious or anything, so it’s not enough to state it and say ‘Kant is wrong’.
The CI is not the claim that everyone should do what you want to do. The CI is the demand (essentially) that you act on reasons. The structure of reasons (like the fact that reasons don’t discriminate between reasoners) gives you the whole ‘universal’ bit.
Oh my, the confusion.
First off, the quoted argument was, as far as I can tell, entirely meant as an illustrative abstraction. The culprit here is the devious function X().
Suppose I take the set of all possible logically coherent statements that could be made about any given mind. Within this set, ‘X’ is any given statement about one mind. X(m) represents whether this given statement is True, False or Undefined / Undecidable for this mind ‘m’.
For all X1..Xn, for a given mind ‘m1’, find all the X that are true. Then for all X() for m2, find those that are true. Supposing any given X has 50% probability of being true of any given m, then X1 being true for m1 has probability 0.5, being true for both m1 and m2 has probability 0.25, and so on dividing the odds by 2 for each additional mind for which the conjunctive must hold true.
So for any given X, for m1..m(10^12), X has (1 / 2^12) probability of being true if we assume a priori 50% chance of that statement being true.
Conversely, for any given X, X(m) will be true for at least one m with 2^12 / (2^12 + 1) probability.
The central inference in the argument is that we do not know the structure of ‘all possible minds’ or of ‘all possible arguments’, but it is reasonable to believe that the space of all possible minds is sufficiently large and versatile that the subset A() of all possible statements X(), where A() are statements of the form “A(m) is true if mind ‘m’, when presented with the argument A, will change some specific belief / internal state / thought / behavior to state Y”, is not true for all A(m). This latter part of the argument rests mostly on the following reasoning:
If there is any given argument A that will convince all currently known minds such that A(m) is true, and all known minds accept the argument, we can almost certainly construct a mind nearly identical to one of these m, but for which the input A is forbidden, or that will self-destruct immediately upon receiving it, or where the first and only possible result of A(m) is always false, or where arguments of the form which A takes are simply incoherent.
For example, if A is a UCMA of the form where you speak or write certain words and have the listener hear or read them, make the listener unable to understand the language. If the UCMA implies translation to a given language of choice that the listener understand, craft the listener so that the listener does not understand or have any languages. If the UCMA is some form of complex hacking by specifying complicated inputs that abuse internal properties of the mind in question… well, how likely is it that all possible minds share these exact internal properties?
This is a complex subject since “minds” and “arguments” are loaded terms with lots of anthropomorphic stuff hidden behind them, and much of the actual meat of the problem lies in things philosophers still disagree on despite overwhelming probability.
Excellently explained, thank you. The argument you present seems to me to be on the whole reasonable, but it involves two assumptions no UCMA enthusiast I know of would ever accept.
And
These two assumptions aren’t argued for, nor are they attributed to any UCMA enthusiast, so I cant see any reason why she should accept them. Do they seem plausible to you? If so, can you give me reasons to accept them?
This isn’t a direct claim of fact, but a flat assumption to simplify illustration and calculations. The same argument extends for any arbitrary probability by showing that mathematically, no matter the probability in question (as long as it is a probability, and not a 0 or 1), as the number of possible minds grows towards infinity, the chance of X being true for all minds keeps decreasing in a similar manner.
The hidden assumption behind this, of course, is that I have a high prior that the number of different possible minds is sufficiently high and the probability comparably low enough for this compound probabilistic growth to become critical. Since the number of known different minds already exceeds seven billion and any given random statement of the form “Mind ‘m’ believes that X is immoral under Y circumstances or context Z” is extremely unlikely to be true for any given mind (and like above, scales to nigh-infinitesimal when conjugating it across all seven billion), I think this hidden assumption is a very reasonable one. An example for clarity:
Mind M ( John B Gato ) believes ( Looking at Jello ) is immoral in context ( Five days and seven minutes after every new moon for a period of three hours, or while scratching one’s toe. )
Of course, this is an intuition about moral beliefs, not about being-convinced-by-arguments, but it’s an intuition about the diversity of ways human minds process arguments that hints at the possible diversity among non-human mind structures.
This is my own model / abstraction of Argument—Mind—Belief/Action. If a UCMA supporter does not believe that arguments lead to any change of belief or behavior in a mind once the argument is made to that mind, then that seems to directly contradict the very idea of a universally compelling argument that persuades any mind.
So the quote above is a model for “If the mind is compelled by the argument, it will have a certain property which allows the argument to compell it” (this property may be a simple emergent property of all possible minds that arises from a combination of the basic necessary properties of all minds, or might be implicit in the structure of logic, which I reckon is a main argument for some of the more sophisticated UCMA supporters).
It seems very unlikely to me that a UCMA enthusiast would grant that a UCMA has in any given case only a fifty percent chance of being UC. So to assume this begs the question against them. It may be that the UCMAist is being silly here, or that the burden is on them to show that things are otherwise, but that’s not relevant to the question of the strength of EY’s argument against UCMAs.
It is, but it’s a bit too reasonable: that is, it’s unreasonable to think that the UCMAist actually thinks that the UCMA is already explicitly accepted by everyone, or even that everyone could be immediately or in any circumstances persuaded that the UCMA is true. UCMA’s on this conception are obviously false, but then EY’s argument is wholly trivial. Nor would we need an argument: it is not hard to come up with a single case of moral disagreement, and that’s all that would be necessary. But this would be to attack a stawman.
The UCMAist is committed to some sense being given to the UC bit, you’re right. If we go to an actual UCMAist, like Kant, the explanation looks something like this: People say all sorts of things about their moral beliefs, but no one could have reasons to doubt the UCMA while holding consistant beliefs. This means that in principle, any mind could be persuaded to accept the UCMA, but not any mind under any circumstances. I (Kant) am committed to saying that every mind is so structured that the UCMA is an unavoidable deductive conclusion, not that every mind in every circumstance has or would arrive at the UCMA. So if this is what being a UCMA means:
Then yes, UCMA’s are impossible. But no one has ever thought otherwise, and it remains open whether something very much like them, namely moral arguments which every possible mind is committed to accepting (whether or not they do accept it) is possible.
No no no. The point of the argument is that it doesn’t matter what the probability is. Even if it’s not 50%, the dynamics at work still make us end up with and exponentially small probability that something is universally compelling, just with the raw math.
The burden is on the UCMAist to show that there are structural reasons why minds must necessarily have certain properties that also happen to coincide with the ability to received, understand, and be convinced by arguments, and also coincide with the specific pattern where at least one specific argument will result in the same understanding and the same resulting conviction for all possible minds.
Both of these are a priori extremely unlikely due to certain intuitions about physics and algorithms and due to the mathematical argument Eliezer makes, respectively.
I’d require clarification on what is meant by “committed to accepting” here. They accept the argument and change their beliefs, or they do not accept the argument and do not change their beliefs. For either case, they either do this in all situations or only some situations. They may sometimes accept it and sometimes not accept it.
The Kant formulation you give seems explicitly about humans, only humans and exclusively humans and nothing else. The whole point of EY’s argument against UCMAs is that there are no universally compelling arguments you could make to an AI built in a manner completely alien to humans that would convince the AI that it is wrong to burn your baby and use its carbon atoms to build more paperclips, even if the AI is fully sentient and capable of producing art and writing philosophy papers about consciousness and universally-compelling moral arguments.
There’s other things I’d say are just wrong about the way this description models minds, but I think that for now I’ll stop here until I’ve read some actual Kant or something.
Right, but I can’t imagine a UMCAist thinking this is a matter of probability. That is, the UMCAist will insist that this is a necessary feature of minds. The burden may be up to them, but that’s not EY’s argument (its not an argument against UMCA’s at all). And I took EY to be giving an argument to the effect that UMCA’s are false or at least unlikely. You may be right that EY has successfully argued that if one has no good reasons to believe a UMCA exists, the probability of one existing must be assessed as low. But this isn’t a premise the UMCAist will grant, so I don’t know what work that point could do.
You might be able to argue that, bu that’s not the way Kant sees it. Kant is explicit that this applies to all minds in mind-space (he kind of discovered the idea of mind-space, I think). As to what ‘committed to accepting’ means, you’re right that this needs a lot of working out, working out I haven’t done. Roughly, I mean that one could not have reasons for denying the UMCA while having consistant beliefs. Kant has to argue that it is structural to all possible minds to be unable to entertain an explicit contradiction, but that’s at least a relatively plausible generalization. Still, tall order.
On the whole, I entirely agree with you that a) the burden is on the UCMAist, b) this burden has not been satisfied here or maybe anywhere. I just wanted to raise a concern about EY’s argument in this post, to the effect that it either begs the question against the UCMAist, or that it is invalid (depending on how it’s interpreted). The shortcomings of the UCMAist aren’t strictly relevant to the (alleged) shortcomings of EY’s anti-UCMAist argument.