OK, I’ve read the whole FAQ. Clearly, a really detailed critique would have to be given at similar length. Therefore, here is just a sketch of the problems I see with your exposition.
For start, you use several invalid examples, or at least controversial examples that you incorrectly present as clear-cut. For example, the phlogiston theory was nothing like the silly strawman you present. It was a falsifiable scientific theory that was abandoned because it was eventually falsified (when it was discovered that burning stuff adds mass due to oxidation, rather than losing mass due to escaped phlogiston). It was certainly a reductionist theory—it attempted to reduce fire (which itself has different manifestations) and the human and animal metabolism to the same underlying physical process. (Google “Becher-Stahl theory”.) Or, at another place, you present the issue of “opposing condoms” as a clear-cut case of “a horrendous decision” from a consequentialist perspective—although in reality the question is far less clear.
Otherwise, up to Section 4, your argumentation is passable. But then it goes completely off the rails. I’ll list just a few main issues:
In the discussion of the trolley problem, you present a miserable caricature of the “don’t push” arguments. The real reason why pushing the fat main is problematic requires delving into a broader game-theoretic analysis that establishes the Schelling points that hold in interactions between people, including those gravest ones that define unprovoked deadly assault. The reason why any sort of organized society is possible is that you can trust that other people will always respect these Schelling points without regards to any cost-benefit calculations, except perhaps when the alternative to violating them is by orders of magnitude more awful than in the trolley examples. (I have compressed an essay’s worth of arguments into a few sentences, but I hope the main point is clear.)
In Section 5, you don’t even mention the key problem of how utilities are supposed to be compared and aggregated interpersonally. If you cannot address this issue convincingly, the whole edifice crumbles.
In Section 6, at first it seems like you get the important point that even if we agree on some aggregate welfare maximization, we have no hope of getting any practical guidelines for action beyond quasi-deontologist heuristics. But they you boldly declare that “we do have procedures in place for breaking the heuristic when we need to.” No, we don’t. You may think we have them, but what we actually have are either somewhat more finely tuned heuristics that aren’t captured by simple first-order formulations (which is good), or rationalizations and other nonsensical arguments couched in terms of a plausible-sounding consequentialist analysis (which is often a recipe for disaster). The law of unintended consequences often bites even in seemingly clear-cut “what could possibly go wrong?” situations.
Along similar lines, you note that in any conflict all parties are quick to point out that their natural rights are at stake. Well, guess what. If they just have smart enough advocates, they can also all come up with different consequentialist analyses whose implications favor their interests. Different ways of interpersonal utility comparison are often themselves enough to tilt the scales as you like. Further, these analyses will all by necessity be based on spherical-cow models of the real world, which you can usually engineer to get pretty much any implication you like.
Section 7 is rather incoherent. You jump from one case study to another arguing that even when it seems like consequentialism might imply something revolting, that’s not really so. Well, if you’re ready to bite awful consequentialist bullets like Robin Hanson does, then be explicit about it. Otherwise, clarify where exactly you draw the lines.
Since we’re already at biting bullets, your FAQ fails to address another crucial issue: it is normal for humans to value the welfare of some people more than others. You clearly value your own welfare and the welfare of your family and friends more than strangers (and even for strangers there are normally multiple circles of diminishing caring). How to reconcile this with global maximization of aggregate utility? Or do you bite the bullet that it’s immoral to care about one’s own family and friends more than strangers?
Question 7.6 is the only one where you give even a passing nod to game-theoretical issues. Considering their fundamental importance in the human social order and all human interactions, and their complex and often counter-intuitive nature, this fact by itself means that most of your discussion is likely to be remote from reality. This is another aspect of the law of unintended consequences that you nonchalantly ignore.
Finally, your idea that it is possible to employ economists and statisticians and get accurate and objective consequentialist analysis to guide public policy is altogether utopian. If such things were possible, economic central planning would be a path to prosperity, not the disaster that it is. (That particular consequentialist folly was finally abandoned in the mainstream after it had produced utter disaster in a sizable part of the world, but many currently fashionable ideas about “scientific” management of government and society suffer from similar delusions.)
Phlogiston: my only knowledge of the theory is Eliezer’s posts on it. Do Eliezer’s posts make the same mistake, or am I misunderstanding those posts?
Trolley-problem: Agreed about Schelling points of interactions between people. What I am trying to do is not make a case for pushing people in hypothetical trolley problems, but to show that certain arguments against doing so are wrong. I think I returned to some of the complicating factors later on, although I didn’t go quite so deep as to mention Schelling points by name. I’ll look through it again and make sure I’ve covered that to at least the low level that would be expected in an introductory argument like this.
Aggregating interpersonal utilities: Admitted that I handwave this away by saying “Economists have some ideas on how to do this”. The FAQ was never meant to get technical, only provide an introduction to the subject. Because it is already 25 pages long I don’t want to go that deep, although I should definitely make it much clearer that these topics exist.
Procedures in place for violating heuristics: By this I mean that we have laws that sometimes supervene certain rights. For one example, even though we have a right to free speech, we also have a law against hate speech. Even though we have a right to property, we also have laws of eminent domain when one piece of property is blocking construction of a railway or something. Would it be proper to rephrase your objection as “We don’t have a single elegant philosophical rule for deciding when it is or isn’t okay to violate heuristics”?
Parties pointing out natural rights are at stake: In a deontological system, these conflicts are not solveable even in principle: we simply don’t know how to decide between two different rights and the only hope is to refer it to politicians or the electorate or philosophers. In a consequentialist system it’s certainly possible to disagree, and clever arguers can come up with models in their favor, but it’s possible to develop mathematical and scientific tools for solving the problem (for example, prediction markets would solve half of this and serious experimental philosophy could make a dent on the other half). And there are certain problems which are totally opaque to rights-based arguments which you couldn’t even begin to argue on consequentialist grounds (eg opt-out organ example given later)
Section 7: I don’t really understand your criticism. Yes, it’s jumping from place to place. I’m answering random objections that people tend to bring up. Do you think I’m straw-manning or missing the important objections? The Nazi and slavery objections at your link seem very much like the racism and slavery objections addressed on the FAQ, and the Hannibal-the-baby-eater objection only seems relevant if one confuses money with utility.
Welfare of some more than others: I admit that I have these preferences, but I don’t think they’re moral preferences. I might choose to save my mother rather than two strangers, I just would be doing it for reasons other than morality. This strikes me as a really weird objection—is there some large group of people who say that nepotism is the moral thing to do?
Game theoretic issues: Agreed that these are important. This is meant to be an introductory FAQ to prime some intuitions, not a complete description of all human behavior. Given that game theory usually means that consequentialism is more likely to give the intuitively correct answer to moral dilemmas, I don’t feel like I’m being dishonest or cherry-picking by excluding most mentions of it. (game theory is against consequentialism only if you mistake consequentialism for certain consequentialism-signaling actions, like pushing people in front of trolleys or assassinating Hitler, rather than considering it as the thought process generating these actions. Learn the thought process first, then master the caveats)
Regarding economists and statisticians: The widespread consensus of economists and statisticians is that economic central planning doesn’t work. I would expect something like prediction markets to be not only be able to guide certain policies, but to be able to accurately predict where to use and where not to use prediction markets.
General response to your comments: Mostly right but too deep for the level at which this FAQ is intended. I will try to revise the FAQ to emphasize that the FAQ is intended only to teach consequentialist thought processes, and that these must then be modified by knowledge of things like game theory.
Each of these issues could be the subject of a separate lengthy discussion, but I’ll try to address them as succinctly as possible:
Re: phlogiston. Yes, Eliezer’s account is inaccurate, though it seems like you have inadvertently made even more out of it. Generally, one recurring problem in the writings of EY (and various other LW contributors) is that they’re often too quick to proclaim various beliefs and actions as silly and irrational, without adequate fact-checking and analysis.
Re: interpersonal utility aggregation/comparison. I don’t think you can handwave this away—it’s a fundamental issue on which everything hinges. For comparison, imagine someone saying that your consequentialism is wrong because it’s contrary to God’s commands, and when you ask how we know that God exists and what his commands are, they handwave it by saying that theologians have some ideas on how to answer these questions. In fact, your appeal to authority is worse in an important sense, since people are well aware that theologians are in disagreement on these issues and have nothing like definite unbiased answers backed by evidence, whereas your answer will leave many people thinking falsely that it’s a well-understood issue where experts can provide adequate answers.
Re: economists and statisticians. Yes, nowadays it’s hard to deny that central planning was a disaster after it crumbled spectacularly everywhere, but read what they were saying before that. Academics are just humans, and if an ideology says that the world is a chaotic inefficient mess and experts like them should be put in charge instead, well, it will be hard for them to resist its pull. Nowadays this folly is finally buried, but a myriad other ones along similar lines are actively being pursued, whose only redeeming value is that they are not as destructive in the short to medium run. (They still make the world uglier and more dysfunctional, and life more joyless and burdensome, in countless ways.) Generally, the idea that you can put experts in charge and expect that they their standards of expertise won’t be superseded by considerations of power and status is naively utopian.
Re: procedures in place for violating heuristics. My problem is not with the lack of elegant philosophical rules. On the contrary, my objections are purely practical. The world is complicated and the law of unintended consequences is merciless and unforgiving. What’s more, humans are scarily good at coming up with seemingly airtight arguments that are in fact pure rationalizations or expressions of intellectual vanity. So, yes, the heuristics must be violated sometimes when the stakes are high enough, but given these realistic limitations, I think you’re way overestimating our ability to identify such situations reliably and the prudence of doing so when the stakes are less than enormous.
Re: Section 7. Basically, you don’t take the least convenient possible world into account. In this case, the LCPW is considering the most awful thing imaginable, assuming that enough people assign it positive enough value that the scales tip in their favor, and then giving a clear answer whether you bite the bullet. Anything less is skirting around the real problem.
Re: welfare of some more than others. I’m confused by your position: are you actually biting the bullet that caring about some people more than others is immoral? I don’t understand why you think it’s weird to ask such a question, since utility maximization is at least prima facie in conflict with both egoism and any sort of preferential altruism, both of which are fundamental to human nature, so it’s unclear how you can resolve this essential problem. In any case, this issue is important and fundamental enough that it definitely should be addressed in your FAQ.
Re: game theory and the thought process. The trouble is that consequentialism, or at least your approach to it, encourages thought processes leading to reckless action based on seemingly sophisticated and logical, but in reality sorely inadequate models and arguments. For example, the idea that you can assess the real-world issue of mass immigration with spherical-cow models like the one to which you link approvingly is every bit as delusional as the idea—formerly as popular among economists as models like this one are nowadays—that you can use their sophisticated models to plan the economy centrally with results far superior to those nasty and messy markets.
General summary: I think your FAQ should at the very least include some discussion of (2) and (6), since these are absolutely fundamental problems. Also, I think you should research more thoroughly the concrete examples you use. If you’ve taken the time to write this FAQ, surely you don’t want people dismissing it because parts of it are inaccurate, even if this isn’t relevant to the main point you’re making.
Regarding the other issues, most of them revolve around the general issues of practical applicability of consequentialist ideas, the law of unintended consequences (of which game-theoretic complications are just one special case), the reliability of experts when they are in positions where their ideas matter in terms of power, status, and wealth, etc. However you choose to deal with them, I think that even in the most basic discussion of this topic, they deserve more concern than your present FAQ gives them.
I will replace the phlogiston section with something else, maybe along the lines of the example of a medicine putting someone to sleep because it has a “dormitive potency”.
I agree with you that there are lots of complex and messy calculations that stand between consequentialism and correct results, and that at best these are difficult and at worst they are not humanly feasible. However, this idea seems to me fundamentally consequentialist—to make this objection, one starts by assuming consequentialist principles, but then saying they can’t be put into action and so we should retreat from pure consequentialism on consequentialist grounds. The target audience of this FAQ is people who are not even at this level yet—people who don’t even understand that you need to argue against certain “consequentialist” ideas on consequentialist grounds, but rather that they can be dismissed by definition because consequences don’t matter. Someone who accepts consequentialism on a base level but then retreats from it on a higher level is already better informed than the people I am aiming this FAQ at. I will make this clearer.
This gets into the political side of things as well. I still don’t understand why you think consequentialism implies or even suggests centralized economic planning when we both agree centralized economic planning would have bad consequences. Certain decisions have to be made, and making them on consequentialist grounds will produce the best results—even if those consequentialist grounds are “never give the government the power to make these decisions because they will screw them up and that will have bad consequences”. I continue to think prediction markets allow something slightly more interesting than that, and I think if you disagree we can resolve that disagreement only on consequentialist grounds—eg would a government that tried to intervene where prediction markets recommended intervention create better consequences than one that didn’t. Nevertheless, I’ll probably end up deleting a lot of this section since it seemed to give everyone an impression I don’t endorse.
Hopefully the changes I listed in my other comment on this thread should help with some of your other worries.
However, this idea seems to me fundamentally consequentialist—to make this objection, one starts by assuming consequentialist principles, but then saying they can’t be put into action and so we should retreat from pure consequentialism on consequentialist grounds.
Fair enough. Though I can grant this only for consequentialism in general, not utilitarianism—unless you have a solution to the fundamental problem of interpersonal utility comparison and aggregation. (In which case I’d be extremely curious to hear it.)
I still don’t understand why you think consequentialism implies or even suggests centralized economic planning when we both agree centralized economic planning would have bad consequences.
I gave it as a historical example of a once wildly popular bad idea that was a product of consequentialist thinking. Of course, as you point out, that was an instance of flawed consequentialist thinking, since the consequences were in fact awful. The problem however is that these same patterns of thinking are by no means dead and gone—it is only that some of their particular instances have been so decisively discredited in practice that nobody serious supports them any more. (And in many other instances, gross failures are still being rationalized away.)
The patterns of thinking I have in mind are more or less what you yourself propose as a seemingly attractive consequentialist approach to problems of public concern: let’s employ accredited experts who will use their sophisticated models to do a cost-benefit analysis and figure out a welfare-maximizing policy. Yes, this really sounds much more rational and objective compared to resolving issues via traditional customs and institutions, which appear to be largely antiquated, irrational, and arbitrary. It also seems far more rational than debating issues in terms of metaphysical constructs such as “liberties,” “rights,” “justice,” “constitutionality,” etc. Trouble is, with very few exceptions, it is usually a recipe for disaster.
Traditional institutions and metaphysical decision-making heuristics are far from perfect, but with a bit of luck, at least they can provide for a functional society. They are a product of cultural (and to some degree biological) evolution, as as such they are quite robust against real-world problems. In contrast, the experts’ models will sooner or later turn out to be flawed one way or another—the difficulty of the problems and the human biases that immediately rear their heads as soon as power and status are at stake practically guarantee this outcome.
Ultimately, when science is used to create policy, the practical outcome is that official science will be debased and corrupted to make it conform to ideological and political pressures. It will not result in elevation of public discourse to a real scientific standard (what you call reducing politics to math) -- that is an altogether utopian idea. So, for example, when that author whose article you linked uses sophisticated-looking math to “analyze” a controversial political issue (in this case immigration), he’s not bringing mathematical clarity and precision of thought into the public discourse. Rather, he is debasing science by concocting a shoddy spherical-cow model with no connection to reality that has some superficial trappings of scientific discourse; the end product is nothing more than Dark Arts. Of course, that was just a blog post, but the situation with real accredited expert output is often not much better.
Now, you can say that I have in fact been making a consequentialist argument all along. In some sense, I agree, but what I wrote certainly applies even to the minimalist interpretation of your positions stated in the FAQ.
Applying these insights to the fat man/trolley problem, we see that the horrible thing about pushing the man is that it transgresses the gravest and most terrible Schelling point of all: the one that defines unprovoked deadly assault, whose violation is understood to give the other party the licence to kill the violator in self-defense. Normally, humans see such crucial Schelling points as sacrosanct. They are considered violable, if at all, only if the consequentialist scales are loaded to a far more extreme degree than in the common trolley problem formulations. Even in the latter case, the act will likely cause serious psychological damage. This is probably an artifact of additional commitment not to violate them, which may also be a safeguard against rationalizations.
Now, the utilitarian may reply that this is just human bias, an unfortunate artifact of evolutionary psychology, and we’d all be better off if people instead made decisions according to pure utilitarian calculus. However, even ignoring all the other fatal problems of utilitarianism, this view is utterly myopic. Humans are able to coordinate and cooperate because we pay respect to the Schelling points (almost) no matter what, and we can trust that others will also do so. If this were not so, you would have to be constantly alert that anyone might rob, kill, cheat, or injure you at any moment because their cost-benefit calculations have implied doing so, even if these calcualtions were in terms of the most idealistic altruistic utilitarianism. Clearly, no organized society could exist in that case: even if with unlimited computational power and perfect strategic insight you could compute that cooperation is viable, this would clearly be impractical.
It is however possible in practice for humans to evaluate each other’s personalities and figure out if others’, so to say, decision algorithms follow these constraints. Think of how people react when they realize that someone has a criminal history or sociopathic tendencies. This person is immediately perceived as creepy and dangerous, and with good reason: people realize that his decision algorithm lacks respect for the conventional Schelling points, so that normal trust and relaxed cooperation with him is impossible, and one must be on the lookout for nasty surprises. Similarly, imagine meeting someone who was in the fat man/trolley situation and who mechanically made the utilitarian decision and pushed the man without a twitch of guilt. Even the most zealous utilitarian will in practice be creeped out by such a person, even though he should theoretically perceive him as an admirable hero. (As always when it comes to ideology, people may be big on words but usually know better when their own welfare is at stake.)
(This comment is also cursory and simplified, and an alert reader will likely catch multiple imprecisions and oversimplifications. This is unfortunately unavoidable because of the complexity of the topic. However, the main point stands regardless. In particular, I haven’t addressed the all too common cases where cooperation between people breaks down and all sorts of conflict ensue. But this analysis would just reinforce the main point that cooperation critically depends on mutual recognition of near-unconditional respect for Schelling points.)
The switch example is more difficult to analyze in terms of the intuitions it evokes. I would guess that the principle of double effect captures an important aspect of what’s going on, though I’m not sure how exactly. I don’t claim to have anything close to a complete theory of human moral intuitions.
In any case, the fact that someone who flipped the switch appears much less (if at all) bad compared to someone who pushed the fat man does suggest strongly that there is some important game-theoretic issue involved, or otherwise we probably wouldn’t have evolved such an intuition (either culturally or genetically). In my view, this should be the starting point for studying these problems, with humble recognition that we are still largely ignorant about how humans actually manage to cooperate and coordinate their actions, instead of naive scoffing at how supposedly innumerate and inconsistent our intuitions are.
OK, I’ve read the whole FAQ. Clearly, a really detailed critique would have to be given at similar length. Therefore, here is just a sketch of the problems I see with your exposition.
For start, you use several invalid examples, or at least controversial examples that you incorrectly present as clear-cut. For example, the phlogiston theory was nothing like the silly strawman you present. It was a falsifiable scientific theory that was abandoned because it was eventually falsified (when it was discovered that burning stuff adds mass due to oxidation, rather than losing mass due to escaped phlogiston). It was certainly a reductionist theory—it attempted to reduce fire (which itself has different manifestations) and the human and animal metabolism to the same underlying physical process. (Google “Becher-Stahl theory”.) Or, at another place, you present the issue of “opposing condoms” as a clear-cut case of “a horrendous decision” from a consequentialist perspective—although in reality the question is far less clear.
Otherwise, up to Section 4, your argumentation is passable. But then it goes completely off the rails. I’ll list just a few main issues:
In the discussion of the trolley problem, you present a miserable caricature of the “don’t push” arguments. The real reason why pushing the fat main is problematic requires delving into a broader game-theoretic analysis that establishes the Schelling points that hold in interactions between people, including those gravest ones that define unprovoked deadly assault. The reason why any sort of organized society is possible is that you can trust that other people will always respect these Schelling points without regards to any cost-benefit calculations, except perhaps when the alternative to violating them is by orders of magnitude more awful than in the trolley examples. (I have compressed an essay’s worth of arguments into a few sentences, but I hope the main point is clear.)
In Section 5, you don’t even mention the key problem of how utilities are supposed to be compared and aggregated interpersonally. If you cannot address this issue convincingly, the whole edifice crumbles.
In Section 6, at first it seems like you get the important point that even if we agree on some aggregate welfare maximization, we have no hope of getting any practical guidelines for action beyond quasi-deontologist heuristics. But they you boldly declare that “we do have procedures in place for breaking the heuristic when we need to.” No, we don’t. You may think we have them, but what we actually have are either somewhat more finely tuned heuristics that aren’t captured by simple first-order formulations (which is good), or rationalizations and other nonsensical arguments couched in terms of a plausible-sounding consequentialist analysis (which is often a recipe for disaster). The law of unintended consequences often bites even in seemingly clear-cut “what could possibly go wrong?” situations.
Along similar lines, you note that in any conflict all parties are quick to point out that their natural rights are at stake. Well, guess what. If they just have smart enough advocates, they can also all come up with different consequentialist analyses whose implications favor their interests. Different ways of interpersonal utility comparison are often themselves enough to tilt the scales as you like. Further, these analyses will all by necessity be based on spherical-cow models of the real world, which you can usually engineer to get pretty much any implication you like.
Section 7 is rather incoherent. You jump from one case study to another arguing that even when it seems like consequentialism might imply something revolting, that’s not really so. Well, if you’re ready to bite awful consequentialist bullets like Robin Hanson does, then be explicit about it. Otherwise, clarify where exactly you draw the lines.
Since we’re already at biting bullets, your FAQ fails to address another crucial issue: it is normal for humans to value the welfare of some people more than others. You clearly value your own welfare and the welfare of your family and friends more than strangers (and even for strangers there are normally multiple circles of diminishing caring). How to reconcile this with global maximization of aggregate utility? Or do you bite the bullet that it’s immoral to care about one’s own family and friends more than strangers?
Question 7.6 is the only one where you give even a passing nod to game-theoretical issues. Considering their fundamental importance in the human social order and all human interactions, and their complex and often counter-intuitive nature, this fact by itself means that most of your discussion is likely to be remote from reality. This is another aspect of the law of unintended consequences that you nonchalantly ignore.
Finally, your idea that it is possible to employ economists and statisticians and get accurate and objective consequentialist analysis to guide public policy is altogether utopian. If such things were possible, economic central planning would be a path to prosperity, not the disaster that it is. (That particular consequentialist folly was finally abandoned in the mainstream after it had produced utter disaster in a sizable part of the world, but many currently fashionable ideas about “scientific” management of government and society suffer from similar delusions.)
Phlogiston: my only knowledge of the theory is Eliezer’s posts on it. Do Eliezer’s posts make the same mistake, or am I misunderstanding those posts?
Trolley-problem: Agreed about Schelling points of interactions between people. What I am trying to do is not make a case for pushing people in hypothetical trolley problems, but to show that certain arguments against doing so are wrong. I think I returned to some of the complicating factors later on, although I didn’t go quite so deep as to mention Schelling points by name. I’ll look through it again and make sure I’ve covered that to at least the low level that would be expected in an introductory argument like this.
Aggregating interpersonal utilities: Admitted that I handwave this away by saying “Economists have some ideas on how to do this”. The FAQ was never meant to get technical, only provide an introduction to the subject. Because it is already 25 pages long I don’t want to go that deep, although I should definitely make it much clearer that these topics exist.
Procedures in place for violating heuristics: By this I mean that we have laws that sometimes supervene certain rights. For one example, even though we have a right to free speech, we also have a law against hate speech. Even though we have a right to property, we also have laws of eminent domain when one piece of property is blocking construction of a railway or something. Would it be proper to rephrase your objection as “We don’t have a single elegant philosophical rule for deciding when it is or isn’t okay to violate heuristics”?
Parties pointing out natural rights are at stake: In a deontological system, these conflicts are not solveable even in principle: we simply don’t know how to decide between two different rights and the only hope is to refer it to politicians or the electorate or philosophers. In a consequentialist system it’s certainly possible to disagree, and clever arguers can come up with models in their favor, but it’s possible to develop mathematical and scientific tools for solving the problem (for example, prediction markets would solve half of this and serious experimental philosophy could make a dent on the other half). And there are certain problems which are totally opaque to rights-based arguments which you couldn’t even begin to argue on consequentialist grounds (eg opt-out organ example given later)
Section 7: I don’t really understand your criticism. Yes, it’s jumping from place to place. I’m answering random objections that people tend to bring up. Do you think I’m straw-manning or missing the important objections? The Nazi and slavery objections at your link seem very much like the racism and slavery objections addressed on the FAQ, and the Hannibal-the-baby-eater objection only seems relevant if one confuses money with utility.
Welfare of some more than others: I admit that I have these preferences, but I don’t think they’re moral preferences. I might choose to save my mother rather than two strangers, I just would be doing it for reasons other than morality. This strikes me as a really weird objection—is there some large group of people who say that nepotism is the moral thing to do?
Game theoretic issues: Agreed that these are important. This is meant to be an introductory FAQ to prime some intuitions, not a complete description of all human behavior. Given that game theory usually means that consequentialism is more likely to give the intuitively correct answer to moral dilemmas, I don’t feel like I’m being dishonest or cherry-picking by excluding most mentions of it. (game theory is against consequentialism only if you mistake consequentialism for certain consequentialism-signaling actions, like pushing people in front of trolleys or assassinating Hitler, rather than considering it as the thought process generating these actions. Learn the thought process first, then master the caveats)
Regarding economists and statisticians: The widespread consensus of economists and statisticians is that economic central planning doesn’t work. I would expect something like prediction markets to be not only be able to guide certain policies, but to be able to accurately predict where to use and where not to use prediction markets.
General response to your comments: Mostly right but too deep for the level at which this FAQ is intended. I will try to revise the FAQ to emphasize that the FAQ is intended only to teach consequentialist thought processes, and that these must then be modified by knowledge of things like game theory.
Each of these issues could be the subject of a separate lengthy discussion, but I’ll try to address them as succinctly as possible:
Re: phlogiston. Yes, Eliezer’s account is inaccurate, though it seems like you have inadvertently made even more out of it. Generally, one recurring problem in the writings of EY (and various other LW contributors) is that they’re often too quick to proclaim various beliefs and actions as silly and irrational, without adequate fact-checking and analysis.
Re: interpersonal utility aggregation/comparison. I don’t think you can handwave this away—it’s a fundamental issue on which everything hinges. For comparison, imagine someone saying that your consequentialism is wrong because it’s contrary to God’s commands, and when you ask how we know that God exists and what his commands are, they handwave it by saying that theologians have some ideas on how to answer these questions. In fact, your appeal to authority is worse in an important sense, since people are well aware that theologians are in disagreement on these issues and have nothing like definite unbiased answers backed by evidence, whereas your answer will leave many people thinking falsely that it’s a well-understood issue where experts can provide adequate answers.
Re: economists and statisticians. Yes, nowadays it’s hard to deny that central planning was a disaster after it crumbled spectacularly everywhere, but read what they were saying before that. Academics are just humans, and if an ideology says that the world is a chaotic inefficient mess and experts like them should be put in charge instead, well, it will be hard for them to resist its pull. Nowadays this folly is finally buried, but a myriad other ones along similar lines are actively being pursued, whose only redeeming value is that they are not as destructive in the short to medium run. (They still make the world uglier and more dysfunctional, and life more joyless and burdensome, in countless ways.) Generally, the idea that you can put experts in charge and expect that they their standards of expertise won’t be superseded by considerations of power and status is naively utopian.
Re: procedures in place for violating heuristics. My problem is not with the lack of elegant philosophical rules. On the contrary, my objections are purely practical. The world is complicated and the law of unintended consequences is merciless and unforgiving. What’s more, humans are scarily good at coming up with seemingly airtight arguments that are in fact pure rationalizations or expressions of intellectual vanity. So, yes, the heuristics must be violated sometimes when the stakes are high enough, but given these realistic limitations, I think you’re way overestimating our ability to identify such situations reliably and the prudence of doing so when the stakes are less than enormous.
Re: Section 7. Basically, you don’t take the least convenient possible world into account. In this case, the LCPW is considering the most awful thing imaginable, assuming that enough people assign it positive enough value that the scales tip in their favor, and then giving a clear answer whether you bite the bullet. Anything less is skirting around the real problem.
Re: welfare of some more than others. I’m confused by your position: are you actually biting the bullet that caring about some people more than others is immoral? I don’t understand why you think it’s weird to ask such a question, since utility maximization is at least prima facie in conflict with both egoism and any sort of preferential altruism, both of which are fundamental to human nature, so it’s unclear how you can resolve this essential problem. In any case, this issue is important and fundamental enough that it definitely should be addressed in your FAQ.
Re: game theory and the thought process. The trouble is that consequentialism, or at least your approach to it, encourages thought processes leading to reckless action based on seemingly sophisticated and logical, but in reality sorely inadequate models and arguments. For example, the idea that you can assess the real-world issue of mass immigration with spherical-cow models like the one to which you link approvingly is every bit as delusional as the idea—formerly as popular among economists as models like this one are nowadays—that you can use their sophisticated models to plan the economy centrally with results far superior to those nasty and messy markets.
General summary: I think your FAQ should at the very least include some discussion of (2) and (6), since these are absolutely fundamental problems. Also, I think you should research more thoroughly the concrete examples you use. If you’ve taken the time to write this FAQ, surely you don’t want people dismissing it because parts of it are inaccurate, even if this isn’t relevant to the main point you’re making.
Regarding the other issues, most of them revolve around the general issues of practical applicability of consequentialist ideas, the law of unintended consequences (of which game-theoretic complications are just one special case), the reliability of experts when they are in positions where their ideas matter in terms of power, status, and wealth, etc. However you choose to deal with them, I think that even in the most basic discussion of this topic, they deserve more concern than your present FAQ gives them.
Okay, thank you.
I will replace the phlogiston section with something else, maybe along the lines of the example of a medicine putting someone to sleep because it has a “dormitive potency”.
I agree with you that there are lots of complex and messy calculations that stand between consequentialism and correct results, and that at best these are difficult and at worst they are not humanly feasible. However, this idea seems to me fundamentally consequentialist—to make this objection, one starts by assuming consequentialist principles, but then saying they can’t be put into action and so we should retreat from pure consequentialism on consequentialist grounds. The target audience of this FAQ is people who are not even at this level yet—people who don’t even understand that you need to argue against certain “consequentialist” ideas on consequentialist grounds, but rather that they can be dismissed by definition because consequences don’t matter. Someone who accepts consequentialism on a base level but then retreats from it on a higher level is already better informed than the people I am aiming this FAQ at. I will make this clearer.
This gets into the political side of things as well. I still don’t understand why you think consequentialism implies or even suggests centralized economic planning when we both agree centralized economic planning would have bad consequences. Certain decisions have to be made, and making them on consequentialist grounds will produce the best results—even if those consequentialist grounds are “never give the government the power to make these decisions because they will screw them up and that will have bad consequences”. I continue to think prediction markets allow something slightly more interesting than that, and I think if you disagree we can resolve that disagreement only on consequentialist grounds—eg would a government that tried to intervene where prediction markets recommended intervention create better consequences than one that didn’t. Nevertheless, I’ll probably end up deleting a lot of this section since it seemed to give everyone an impression I don’t endorse.
Hopefully the changes I listed in my other comment on this thread should help with some of your other worries.
Fair enough. Though I can grant this only for consequentialism in general, not utilitarianism—unless you have a solution to the fundamental problem of interpersonal utility comparison and aggregation. (In which case I’d be extremely curious to hear it.)
I gave it as a historical example of a once wildly popular bad idea that was a product of consequentialist thinking. Of course, as you point out, that was an instance of flawed consequentialist thinking, since the consequences were in fact awful. The problem however is that these same patterns of thinking are by no means dead and gone—it is only that some of their particular instances have been so decisively discredited in practice that nobody serious supports them any more. (And in many other instances, gross failures are still being rationalized away.)
The patterns of thinking I have in mind are more or less what you yourself propose as a seemingly attractive consequentialist approach to problems of public concern: let’s employ accredited experts who will use their sophisticated models to do a cost-benefit analysis and figure out a welfare-maximizing policy. Yes, this really sounds much more rational and objective compared to resolving issues via traditional customs and institutions, which appear to be largely antiquated, irrational, and arbitrary. It also seems far more rational than debating issues in terms of metaphysical constructs such as “liberties,” “rights,” “justice,” “constitutionality,” etc. Trouble is, with very few exceptions, it is usually a recipe for disaster.
Traditional institutions and metaphysical decision-making heuristics are far from perfect, but with a bit of luck, at least they can provide for a functional society. They are a product of cultural (and to some degree biological) evolution, as as such they are quite robust against real-world problems. In contrast, the experts’ models will sooner or later turn out to be flawed one way or another—the difficulty of the problems and the human biases that immediately rear their heads as soon as power and status are at stake practically guarantee this outcome.
Ultimately, when science is used to create policy, the practical outcome is that official science will be debased and corrupted to make it conform to ideological and political pressures. It will not result in elevation of public discourse to a real scientific standard (what you call reducing politics to math) -- that is an altogether utopian idea. So, for example, when that author whose article you linked uses sophisticated-looking math to “analyze” a controversial political issue (in this case immigration), he’s not bringing mathematical clarity and precision of thought into the public discourse. Rather, he is debasing science by concocting a shoddy spherical-cow model with no connection to reality that has some superficial trappings of scientific discourse; the end product is nothing more than Dark Arts. Of course, that was just a blog post, but the situation with real accredited expert output is often not much better.
Now, you can say that I have in fact been making a consequentialist argument all along. In some sense, I agree, but what I wrote certainly applies even to the minimalist interpretation of your positions stated in the FAQ.
I unfortunately don’t get the main point :(
Could you elaborate on or at least provide a reference for how a consideration of Schelling points would suggest that we shouldn’t push the fat man?
This essay by David Friedman is probably the best treatment of the subject of Schelling points in human relations:
http://www.daviddfriedman.com/Academic/Property/Property.html
Applying these insights to the fat man/trolley problem, we see that the horrible thing about pushing the man is that it transgresses the gravest and most terrible Schelling point of all: the one that defines unprovoked deadly assault, whose violation is understood to give the other party the licence to kill the violator in self-defense. Normally, humans see such crucial Schelling points as sacrosanct. They are considered violable, if at all, only if the consequentialist scales are loaded to a far more extreme degree than in the common trolley problem formulations. Even in the latter case, the act will likely cause serious psychological damage. This is probably an artifact of additional commitment not to violate them, which may also be a safeguard against rationalizations.
Now, the utilitarian may reply that this is just human bias, an unfortunate artifact of evolutionary psychology, and we’d all be better off if people instead made decisions according to pure utilitarian calculus. However, even ignoring all the other fatal problems of utilitarianism, this view is utterly myopic. Humans are able to coordinate and cooperate because we pay respect to the Schelling points (almost) no matter what, and we can trust that others will also do so. If this were not so, you would have to be constantly alert that anyone might rob, kill, cheat, or injure you at any moment because their cost-benefit calculations have implied doing so, even if these calcualtions were in terms of the most idealistic altruistic utilitarianism. Clearly, no organized society could exist in that case: even if with unlimited computational power and perfect strategic insight you could compute that cooperation is viable, this would clearly be impractical.
It is however possible in practice for humans to evaluate each other’s personalities and figure out if others’, so to say, decision algorithms follow these constraints. Think of how people react when they realize that someone has a criminal history or sociopathic tendencies. This person is immediately perceived as creepy and dangerous, and with good reason: people realize that his decision algorithm lacks respect for the conventional Schelling points, so that normal trust and relaxed cooperation with him is impossible, and one must be on the lookout for nasty surprises. Similarly, imagine meeting someone who was in the fat man/trolley situation and who mechanically made the utilitarian decision and pushed the man without a twitch of guilt. Even the most zealous utilitarian will in practice be creeped out by such a person, even though he should theoretically perceive him as an admirable hero. (As always when it comes to ideology, people may be big on words but usually know better when their own welfare is at stake.)
(This comment is also cursory and simplified, and an alert reader will likely catch multiple imprecisions and oversimplifications. This is unfortunately unavoidable because of the complexity of the topic. However, the main point stands regardless. In particular, I haven’t addressed the all too common cases where cooperation between people breaks down and all sorts of conflict ensue. But this analysis would just reinforce the main point that cooperation critically depends on mutual recognition of near-unconditional respect for Schelling points.)
Can you explain why this analysis renders directing away from the five and toward the one permissible?
The switch example is more difficult to analyze in terms of the intuitions it evokes. I would guess that the principle of double effect captures an important aspect of what’s going on, though I’m not sure how exactly. I don’t claim to have anything close to a complete theory of human moral intuitions.
In any case, the fact that someone who flipped the switch appears much less (if at all) bad compared to someone who pushed the fat man does suggest strongly that there is some important game-theoretic issue involved, or otherwise we probably wouldn’t have evolved such an intuition (either culturally or genetically). In my view, this should be the starting point for studying these problems, with humble recognition that we are still largely ignorant about how humans actually manage to cooperate and coordinate their actions, instead of naive scoffing at how supposedly innumerate and inconsistent our intuitions are.
Thanks! That makes sense.