It’s not that much of a difference. Such a model could still accept that humans are unified individuals, but also attached to parts (defined as not the relevant part of the human) that interfere with the human’s actions.
Roko’s alternative is just to say, “X is that action that I would attempt; hardware inextricably connected to me would also stop me from doing X.”
Of course, that does run into problems like, “So you agree that you’re running on corrupted hardware that stops you from doing what you believe is morally right—why should I trust you, then?
This might make sense, but it breaks down the models of pretty much all standard ethical theories, utilitarian and otherwise, which invariably treat humans as unified individuals.
Except for very narrow definitions of “standard,” this is just incorrect. Plato, Hume, Kant, and John Stuart Mill all understood and wrote about the difference between what they thought of as the rational or refined will and the more emotional appetite. Likewise Maimonides, St. Augustine, Epictetus, and a 16th century Taoist scholar whose name I can look up for you if it’s actually important. In fact, an enormous part of standard ethics deals with the divergence between what we say is right and what we actually do, and tries to identify ways to help us actually do what we say is right.
The blanket assertion that anything you do without being physically restrained is what you wanted to do under the circumstances is a creature of 20th century free-market economics. While it can be part of a self-consistent moral philosophy (e.g. Ayn Rand’s Objectivism), it’s hardly a litmus test for sound ethical thinking. On the contrary, we should be deeply suspicious of any moral theory that tells us that whatever we do must be what we wanted to do, because it conveniently justifies a set of actions that we (apparently) find quite easy to carry out. What is easy is not always right.
I was replying both to you and to Vladimir_M, because both of you seemed to me to be accepting the premise that humans (however defined) must be unitary actors in order to be amenable to coherent ethical accounts.
Understood, but just to be clear, I was only accepting that premise for purposes of argument, saying that you don’t need to resort to non-unitary models to phrase Roko’s position. I don’t accept that premise as a general rule. (Or at least I recognize that this model quickly runs into problems—see my exchange with Vladimir_M.)
Thanks for the link; it’s an interesting dialogue. May I suggest, by way of constructive criticism, that when someone challenges you to play Rationalist’s Taboo, you respond with a formal definition that uses few or no pronouns, regardless of whether you think you have already defined your terms well?
E.g.:
First-order preference (n): a desire for some state X that, if unopposed, usually leads to actions calculated or assumed to bring X about.
Second-order preference (n): a meta-preference; a desire to have some particular ranking R of first-order preferences that, if unopposed, will usually lead to actions calculated or assumed to bring R about.
Akrasia (n): the state of having a first-order preference A that conflicts with a second-order preference B such that A is stronger, and usually wins.
Addiction (n): a subset of akrasia such that, if the person with akrasia were temporarily and forcibly prevented from acting on A, he/she would (1) be grateful AND (2) likely have a reduced preference for A in the future.
Thank you, that’s a great formalism. Under your terminology, my position is that there is a difference between someone with addiction, vs. someone with consistent first/second-order preferences, and that this difference is so empirically significant as to justify having different terms, and that this difference is experimentally detectable (at least in hypothetical situations).
Of course, your definitions define addiction by that experimental difference, and that’s something I’d want to avoid.
Vladimir_M’s point, in turn, is that people with consistent first/second order preferences that are not socially acceptable try to persuade others it is actually a case of addiction in order to increase the net benefit of indulging that preference. I agree with him that this is often what’s going on, but disagree that it can account for all cases, thereby necessitating the distinction of the separate category of akrasia (and addiction).
You are mostly right, except that I disagree that such simplifications are limited to 20th century economics. I had in mind formal ethical theories that I find discussed in modern analytical philosophy, and especially utilitarianism. I honestly don’t see how utilitarianism can make sense unless humans are modeled as unified agents, each with a single utility function. From what I’ve seen, other popular formal consequentialist approaches make analogous assumptions, for which I don’t see how they could be reconciled with dissolving the concept of humans as unified agents.
But yes, considering the vast philosophical tradition you mention, my above statement definitely doesn’t hold in general. However, to get back to the issue that started this discussion, I don’t think that Aspergery logical consistency—that, according to Roko, apparently makes for a good consequentialist ethicist -- would be a good guide through the works of the authors you mention!
Hm. We’re a few levels down from the parent thread here, so please forgive me if I fail to focus on your main point. I’m aiming for it, but I might miss.
It seems like you’re saying that, in so far as we appear to observe a unified human psychology, it may just be because of myopia—upon truly considering a moral dilemma in all its ugly ramifications, people would approve of and adopt different courses of action.
That seems correct as far as it goes, but what if people’s emotions and logic generally follow the same path? What if, upon reflection, all neurotypicals would agree that, ideally speaking, they would like to live in a world where people slit the throats of trolley-obstructors so that light rail would be safe enough to ride on, but each neurotypical individual also agrees that zie could never actually bring zerself to slit an innocent person’s throat, because it would be too yucky?
That still leaves us with the vast array of neuro-atypicals in our search for diversity, but then the question of whether humankind has a unified psychology is still interesting. Instead of the response being an obvious “no, we’re diverse,” the response becomes an investigation of how many atypicals there are, how different their opinions really are, and perhaps which ones are worth viewing as “healthy” enough to count.
Let me just qualify that last remark. I believe there should be room for many different kinds of minds in our society, but that doesn’t mean there’s no such thing as mental illness. For example, a paranoid schizophrenic might have a different opinion about the trolley problem, but I’m not sure we should ask him—maybe we should just offer him some antipsychotics and see if he calms down.
Tangent: The trolley problem actually seems like a relatively innocuous source of diversity. In terms of designing a world we would want to live in I think there is pretty broad agreement that we want our trolleys to not run out of control. Yes, the principles of the trolley controversy could end up leading to disagreement over something far more important but… right now there are people with deep, powerful desires central to their overall happiness, the fulfillment of which other people find morally repugnant and sinful. That strikes me as in many ways a much bigger problem that the deontology/consequentialism battle.
Well, OK, let’s go ahead and flesh that out. I read your Rorschach blot and the first thing that comes to mind is gay sex vs. Christian fundies. Want to run with that for a few minutes? How does it illustrate psychological diversity? Isn’t it just an example of how different beliefs about reality lead to different moral opinions about specific actions? If you could get a Christian fundamentalist to imagine a world where Jesus was just a charismatic preacher and sexual orientation was caused by genes and hormones, wouldn’t she say that gay sex was OK in that world? For that matter, if you could get an openly gay atheist to imagine a world where the New Testament as it has been traditionally interpreted really was the literal word of a God who for some inexplicable reason was so cool that whatever God’s opinions were automatically became morally correct, wouldn’t the gay atheist say that gay sex was sinful in that world? Where’s the gap in human psychology?
Feel free to pick a different example if you had something else in mind. :-)
I was ambiguous because while gay sex and Christian fundamentalism does come to mind so does the entire gamut of pleasurable activities that people object to as wrong or impure. Sex with multiple partners, sex outside of marriage, polygamy, BDSM, homosexuality, paedophilic and ephebophilic fantasy, etc. And it isn’t just Christian fundamentalists doing the condemning either.
Isn’t it just an example of how different beliefs about reality lead to different moral opinions about specific actions?
I don’t know. While the justification given for the opposition to homosexuality is biblical I’m not confident the given justification is the motivation behind the conservative Christian opposition. To me, at least, the Haidt’s concept of moral purity is what is really at work. And this helps explain the revulsion toward a wide range of sexual activities (which may or may not be discussed in the Bible) from people who may or may not have read the Bible.
If you could get a Christian fundamentalist to imagine a world where Jesus was just a charismatic preacher and sexual orientation was caused by genes and hormones, wouldn’t she say that gay sex was OK in that world?
In addition to the above, it seems to me that for many, even most people, religion, morality and sex are all tangled up in the same memetic mess and that such people may not even have a proper map-territory conceptualization of the world. And this entangled collection of memes may not be the direct output of their psychology but I’m not sure any value system is, it is certainly the case that their psychology is extremely amenable to this collection of memes. And it seems very plausible to me that some people have psychologies more amenable to and comfortable with these memes than other people.
And an interesting feature of these oppositions to desires is that they are, at least in part, cultural. It turns out you can turn down or even switch off the disgust instinct to at least some sexual behavior if you raise them right, teach them tolerance and have admirable television characters with these desires.
I’m not sure the case is any different with disagreements in normative theory. Some minds are more amenable to consequentialism, others deontology, others virtue ethics, others are just confused. But there is no reason to think such minds begin like this or are stuck like this. There is no reason to think these theories would be resistant to cultural evolution either (indeed, I’m nearly certain they have been subject to cultural evolution which is why you don’t see consequentialists or liberty-minded deontologists until the Scottish Enlightenment).
Of course, the cultural evolution involved in the increasing acceptance of homosexuality has been very visible and very rapid. A concerted and widespread effort to spread memes conducive to pushing the fat guy onto the tracks probably would change the way people answered that question: but obviously the incentives for spreading that set of memes just aren’t there in the way they are with acceptance of homosexuality. And of course the reason there are no incentives is that consequentialists tend to get along with deontologists just fine and no one gets hurt enough to try and change things.
It’s not that much of a difference. Such a model could still accept that humans are unified individuals, but also attached to parts (defined as not the relevant part of the human) that interfere with the human’s actions.
Roko’s alternative is just to say, “X is that action that I would attempt; hardware inextricably connected to me would also stop me from doing X.”
Of course, that does run into problems like, “So you agree that you’re running on corrupted hardware that stops you from doing what you believe is morally right—why should I trust you, then?
Except for very narrow definitions of “standard,” this is just incorrect. Plato, Hume, Kant, and John Stuart Mill all understood and wrote about the difference between what they thought of as the rational or refined will and the more emotional appetite. Likewise Maimonides, St. Augustine, Epictetus, and a 16th century Taoist scholar whose name I can look up for you if it’s actually important. In fact, an enormous part of standard ethics deals with the divergence between what we say is right and what we actually do, and tries to identify ways to help us actually do what we say is right.
The blanket assertion that anything you do without being physically restrained is what you wanted to do under the circumstances is a creature of 20th century free-market economics. While it can be part of a self-consistent moral philosophy (e.g. Ayn Rand’s Objectivism), it’s hardly a litmus test for sound ethical thinking. On the contrary, we should be deeply suspicious of any moral theory that tells us that whatever we do must be what we wanted to do, because it conveniently justifies a set of actions that we (apparently) find quite easy to carry out. What is easy is not always right.
Was this intended as a reply to the parent of my comment?
I was replying both to you and to Vladimir_M, because both of you seemed to me to be accepting the premise that humans (however defined) must be unitary actors in order to be amenable to coherent ethical accounts.
Understood, but just to be clear, I was only accepting that premise for purposes of argument, saying that you don’t need to resort to non-unitary models to phrase Roko’s position. I don’t accept that premise as a general rule. (Or at least I recognize that this model quickly runs into problems—see my exchange with Vladimir_M.)
Thanks for the link; it’s an interesting dialogue. May I suggest, by way of constructive criticism, that when someone challenges you to play Rationalist’s Taboo, you respond with a formal definition that uses few or no pronouns, regardless of whether you think you have already defined your terms well?
E.g.:
First-order preference (n): a desire for some state X that, if unopposed, usually leads to actions calculated or assumed to bring X about.
Second-order preference (n): a meta-preference; a desire to have some particular ranking R of first-order preferences that, if unopposed, will usually lead to actions calculated or assumed to bring R about.
Akrasia (n): the state of having a first-order preference A that conflicts with a second-order preference B such that A is stronger, and usually wins.
Addiction (n): a subset of akrasia such that, if the person with akrasia were temporarily and forcibly prevented from acting on A, he/she would (1) be grateful AND (2) likely have a reduced preference for A in the future.
Thank you, that’s a great formalism. Under your terminology, my position is that there is a difference between someone with addiction, vs. someone with consistent first/second-order preferences, and that this difference is so empirically significant as to justify having different terms, and that this difference is experimentally detectable (at least in hypothetical situations).
Of course, your definitions define addiction by that experimental difference, and that’s something I’d want to avoid.
Vladimir_M’s point, in turn, is that people with consistent first/second order preferences that are not socially acceptable try to persuade others it is actually a case of addiction in order to increase the net benefit of indulging that preference. I agree with him that this is often what’s going on, but disagree that it can account for all cases, thereby necessitating the distinction of the separate category of akrasia (and addiction).
You are mostly right, except that I disagree that such simplifications are limited to 20th century economics. I had in mind formal ethical theories that I find discussed in modern analytical philosophy, and especially utilitarianism. I honestly don’t see how utilitarianism can make sense unless humans are modeled as unified agents, each with a single utility function. From what I’ve seen, other popular formal consequentialist approaches make analogous assumptions, for which I don’t see how they could be reconciled with dissolving the concept of humans as unified agents.
But yes, considering the vast philosophical tradition you mention, my above statement definitely doesn’t hold in general. However, to get back to the issue that started this discussion, I don’t think that Aspergery logical consistency—that, according to Roko, apparently makes for a good consequentialist ethicist -- would be a good guide through the works of the authors you mention!
Hm. We’re a few levels down from the parent thread here, so please forgive me if I fail to focus on your main point. I’m aiming for it, but I might miss.
It seems like you’re saying that, in so far as we appear to observe a unified human psychology, it may just be because of myopia—upon truly considering a moral dilemma in all its ugly ramifications, people would approve of and adopt different courses of action.
That seems correct as far as it goes, but what if people’s emotions and logic generally follow the same path? What if, upon reflection, all neurotypicals would agree that, ideally speaking, they would like to live in a world where people slit the throats of trolley-obstructors so that light rail would be safe enough to ride on, but each neurotypical individual also agrees that zie could never actually bring zerself to slit an innocent person’s throat, because it would be too yucky?
That still leaves us with the vast array of neuro-atypicals in our search for diversity, but then the question of whether humankind has a unified psychology is still interesting. Instead of the response being an obvious “no, we’re diverse,” the response becomes an investigation of how many atypicals there are, how different their opinions really are, and perhaps which ones are worth viewing as “healthy” enough to count.
Let me just qualify that last remark. I believe there should be room for many different kinds of minds in our society, but that doesn’t mean there’s no such thing as mental illness. For example, a paranoid schizophrenic might have a different opinion about the trolley problem, but I’m not sure we should ask him—maybe we should just offer him some antipsychotics and see if he calms down.
Tangent: The trolley problem actually seems like a relatively innocuous source of diversity. In terms of designing a world we would want to live in I think there is pretty broad agreement that we want our trolleys to not run out of control. Yes, the principles of the trolley controversy could end up leading to disagreement over something far more important but… right now there are people with deep, powerful desires central to their overall happiness, the fulfillment of which other people find morally repugnant and sinful. That strikes me as in many ways a much bigger problem that the deontology/consequentialism battle.
Well, OK, let’s go ahead and flesh that out. I read your Rorschach blot and the first thing that comes to mind is gay sex vs. Christian fundies. Want to run with that for a few minutes? How does it illustrate psychological diversity? Isn’t it just an example of how different beliefs about reality lead to different moral opinions about specific actions? If you could get a Christian fundamentalist to imagine a world where Jesus was just a charismatic preacher and sexual orientation was caused by genes and hormones, wouldn’t she say that gay sex was OK in that world? For that matter, if you could get an openly gay atheist to imagine a world where the New Testament as it has been traditionally interpreted really was the literal word of a God who for some inexplicable reason was so cool that whatever God’s opinions were automatically became morally correct, wouldn’t the gay atheist say that gay sex was sinful in that world? Where’s the gap in human psychology?
Feel free to pick a different example if you had something else in mind. :-)
I was ambiguous because while gay sex and Christian fundamentalism does come to mind so does the entire gamut of pleasurable activities that people object to as wrong or impure. Sex with multiple partners, sex outside of marriage, polygamy, BDSM, homosexuality, paedophilic and ephebophilic fantasy, etc. And it isn’t just Christian fundamentalists doing the condemning either.
I don’t know. While the justification given for the opposition to homosexuality is biblical I’m not confident the given justification is the motivation behind the conservative Christian opposition. To me, at least, the Haidt’s concept of moral purity is what is really at work. And this helps explain the revulsion toward a wide range of sexual activities (which may or may not be discussed in the Bible) from people who may or may not have read the Bible.
In addition to the above, it seems to me that for many, even most people, religion, morality and sex are all tangled up in the same memetic mess and that such people may not even have a proper map-territory conceptualization of the world. And this entangled collection of memes may not be the direct output of their psychology but I’m not sure any value system is, it is certainly the case that their psychology is extremely amenable to this collection of memes. And it seems very plausible to me that some people have psychologies more amenable to and comfortable with these memes than other people.
And an interesting feature of these oppositions to desires is that they are, at least in part, cultural. It turns out you can turn down or even switch off the disgust instinct to at least some sexual behavior if you raise them right, teach them tolerance and have admirable television characters with these desires.
I’m not sure the case is any different with disagreements in normative theory. Some minds are more amenable to consequentialism, others deontology, others virtue ethics, others are just confused. But there is no reason to think such minds begin like this or are stuck like this. There is no reason to think these theories would be resistant to cultural evolution either (indeed, I’m nearly certain they have been subject to cultural evolution which is why you don’t see consequentialists or liberty-minded deontologists until the Scottish Enlightenment).
Of course, the cultural evolution involved in the increasing acceptance of homosexuality has been very visible and very rapid. A concerted and widespread effort to spread memes conducive to pushing the fat guy onto the tracks probably would change the way people answered that question: but obviously the incentives for spreading that set of memes just aren’t there in the way they are with acceptance of homosexuality. And of course the reason there are no incentives is that consequentialists tend to get along with deontologists just fine and no one gets hurt enough to try and change things.
That make sense?
Yes! This is very enlightening; thank you for your thoughtful response. I am convinced, for now. :-)