Yes. The error is that humans aren’t good at utilitarianism.
private_messaging has given an example elsewhere: the trouble with utilitarians is that they think they are utilitarians. They then use numbers to convince themselves to do something they would otherwise consider evil.
The Soviet Union was an attempt to build a Friendly government based on utilitarianism. They quickly reached “shoot someone versus dust specks” and went for shooting people.
They weren’t that good at lesser utilitarian decisions either, tending to ignore how humans actually behaved in favour of taking their theories and shutting-up-and-multiplying. Then when that didn’t work, they did it harder.
I’m sure someone objecting to the Soviet Union example as non-negligible evidence can come up with examples that worked out much better, of course.
The key difference between a normal utilitarian and a Leninist: When a normal utilitarian concludes that mass murder would maximize social utility, he checks his work! He goes over his calculations with a fine-tooth comb, hoping to discover a way to implement beneficial policy changes without horrific atrocities. The Leninist, in contrast, reasons backwards from the atrocities that emotionally inspire him to the utilitarian argument that morally justifies his atrocities.
If this seems woefully uncharitable, compare the amount of time a proto-Leninist like Raskolnikov spends lovingly reviewing the mere conceivability of morally justified bloodbaths to the amount of time he spends (a) empirically evaluating the effects of policies or (b) searching for less brutal ways to implement whatever policies he wants. These ratios are typical for the entire Russian radical tradition; it’s what they imagined to be “profound.” When men like this gained power in Russia, they did precisely what you’d expect: treat mass murder like a panacea. This is the banality of Leninism.
As I have noted, when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to. If arithmetical utilitarianism works so well, it would work in weird territory.
Caplan does have a cultural point on the Soviet Union example. OTOH, it does seem a bit “no true utilitarian”.
If arithmetical utilitarianism works so well, it would work in weird territory.
Note the bank robbery thread below. Someone claims that “the utilitarian math” shows that robbing banks and donating to charity would have the best consequences. But they don’t do any math or look up basic statistics to do a Fermi calculation. A few minutes of effort shows that bank robbery actually pays much worse than working as a bank teller over the course of a career (including jail time, etc).
In Giving What We Can there are several people who donate half their income (or all income above a Western middle class standard of living) to highly efficient charities helping people in the developing world. They expect to donate millions of dollars over their careers, and to have large effects on others through their examples and reputations, both as individuals and via their impact on organizations like Giving What We Can. They do try to actually work things out, and basic calculations easily show that running around stealing organs or robbing banks would have terrible consequences, thanks to strong empirical regularities:
Crime mostly doesn’t pay. Bank robbers, drug dealers, and the like make less than legitimate careers. They also spend a big chunk of time imprisoned, and ruin their employability for the future. Very talented people who might do better than the average criminal can instead go to Wall Street or Silicon Valley and make far more.
Enormous amounts of good can be done through a normal legitimate career. Committing violent crime, or other hated acts close off such opportunities very rapidly.
Really dedicated do-gooders hope to have most of their influence through example, encouraging others to do good. Becoming a hated criminal, and associating their ethical views with such, should be expected to have huge negative effects by staunching the flow of do-gooders to exploit the vast legitimate opportunities to help people.
If some criminal scheme looks easy and low-risk, consider that law enforcement uses many techniques which are not made public, and are very hard for a lone individual to learn. There are honey-pots, confederates, and so forth. In the market for nuclear materials, most of the buyers and sellers are law enforcement agents trying to capture any real criminal participants. In North America terrorist cells are now regularly infiltrated long before they act, with government informants insinuated into the cell, phone and internet activities monitored, etc.
It is hard to keep a crime secret over time. People feel terrible guilt, and often are caught after they confess to others. In the medium term there is some chance of more effective neuroscience-based lie detectors, which goes still higher long-term.
The broader society, over time, could punish utilitarian villainy by reducing its support for the things utilitarians seek as they are associated with villains, or even by producing utilitarian evils. If animal rights terrorists tried to kill off humanity, it might lead to angry people eating more meat or creating anti-utilitronium (by the terrorists’ standards, not so much the broader society, focused on animals, say) in anger. The 9/11 attacks were not good for Osama bin Laden’s ambitions of ruling Saudi Arabia.
There are other considerations, but these are enough to dispense with the vast bestiary of supposedly utility-boosting sorts of wrongdoing. Arithmetical utilitarianism does say you should not try to become a crook. But unstable or vicious people (see the Caplan Leninist link) sometimes do like to take the idea of “the end justifies the means” as an excuse to go commit crimes without even trying to work out how the means are related to the end, and to alternatives.
Disclaimer: I do not value total welfare to the exclusion of other ethical and personal concerns. My moral feelings oppose deontological nastiness aside from aggregate welfare. But I am tired of straw-manning “estimating consequences” and “utilitarian math” by giving examples where these aren’t used and would have prevented the evil conclusion supposedly attributed to them.
I’m confused. Your comment paints a picture of a super-efficient police force that infiltrates criminal groups long before they act. But the Internet seems to say that many gangs in the US operate openly for years, control whole neighborhoods, and have their own Wikipedia pages...
Gang membership still doesn’t pay relative to regular jobs
The police largely know who is in the gangs, and can crack down if this becomes a higher priority
Terrorism is such a priority, to a degree way out of line with the average historical damage, because of 9/11; many have critiqued the diversion of law enforcement resources to terrorism
Such levels of gang control are concentrated in poor areas with less police funding, and areas where the police are estranged from the populace, limiting police activity.
Gang violence is heavily directed at other criminal gangs, reducing the enthusiasm of law enforcement, relative to more photogenic victims
The other side is that robbing banks at gunpoint isn’t the most effective way to redistribute wealth from those who have it to those to whom it should go.
I suspect that the most efficient way to do that is government seizure- declare that the privately held assets of the bank now belong to the charities. That doesn’t work, because the money isn’t value, it’s a signifier of value, and rewriting the map does not change the territory- if money is forcibly redistributed too much, it loses too much value and the only way to enforce the tax collection is by using the threat of prison and execution- but the jailors and executioners can only be paid by the taxes. Effectively robbing banks to give the money to charity harms everyone significantly, and fails to be better than doing nothing.
It may have been better if CarlShulman used a different word—perhaps ‘Evil’ - to represent the ‘ethical injunctions’ idea. That seems to better represent the whole “deliberately subvert consequentialist reasoning in certain areas due to acknowledgement of corrupted and bounded hardware”. ‘Weird’ seems to be exactly the sort of thing Eliezer might advocate. For example “make yourself into a corpsicle” and “donate to SingInst”.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
Of course it does. For example: dressing up as a penguin and beating people to death with a live fish. But that’s largely irrelevant. Rejecting ‘weird’ as the class of things that must never be done is not the same thing as saying that all things in that class must be done. Instead, weirdness is just ignored.
I’ve always felt that post was very suspect. Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal, even if there’s a very low chance of it working. Your own welfare simply doesn’t play a factor, given the size of the variables you’re playing with. It seems to be that there is a deeper moral reason not to murder organ donors or steal food for the hungry than ‘it might end poorly for you.’
Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal
Bank robbery is actually unprofitable. Even setting aside reputation (personal and for one’s ethos), “what if others reasoned similarly,” the negative consequences of the robbery, and so forth you’d generate more expected income working an honest job. This isn’t a coincidence. Bank robbery hurts banks, insurers, and ultimately bank customers, and so they are willing to pay to make it unprofitable.
According to a study by British researchers Barry Reilly, Neil Rickman and Robert Witt written up in this month’s issue of the journal Significance, the average take from a U.S. bank robbery is $4,330. To put that in perspective, PayScale.com says bank tellers can earn as much as $28,205 annually. So, a bank robber would have to knock over more than six banks, facing increasing risk with each robbery, in a year to match the salary of the tellers he’s holding up.
That was a somewhat lazy example, I admit, but consider the most inconvenient possible world. Let’s say you could expect to take a great deal more from a bank robbery. Would it then be valid utilitarian ethics to rob (indirectly) from the rich (us) to give to the poor?
My whole point in the comments on this post has been that it’s a pernicious practice to use such false examples. They leave erroneous impressions and associations. A world where bank-robbery is super-profitable, so profitable as to outweigh the effects of reputation and the like, is not very coherent.
A better example would be something like: “would utilitarians support raising taxes to fund malaria eradication,” or “would a utilitarian who somehow inherited swoopo.com (a dollar auction site) shut down the site or use the revenue to save kids from malaria” or “if a utilitarian inherited the throne in a monarchy like Oman (without the consent of the people) would he spend tax revenues on international good causes or return them to the taxpayers?”
if you do the utilitarian math, robbing banks and giving them to charity is still a good deal
Only if you’re bad at math. Banks aren’t just piggybanks to smash, they perform a useful function in the economy, and to disrupt it has consequences.
Of course I prefer to defeat bad utilitarian math with better utilitarian math rather than with ethical injunctions. But hey, that’s the woe of bounded reason, even without going into the whole corrupted hardware problem: your model is only so good, and heuristics that serve as warning signals have their place.
Yes. The error is that humans aren’t good at utilitarianism.
Why would that be an error? It’s not a requirement for an ethical theory that Homo sapiens must be good at it. If we notice that humans are bad at it, maybe we should make AI or posthumans that are better at it, if we truly view this as the best ethical theory. Besides, if the outcome of people following utilitarianism is really that bad, then utilitarianism would demand (it gets meta now) that people should follow some other theory that overall has better outcomes (see also Parfit’s Reasons and Persons). Another solution is Hare’s proposed “Two-Level Utilitarianism”.
From Wikipedia:
Hare proposed that on a day to day basis, one should think and act like a rule utilitarian and follow a set of intuitive prima facie rules, in order to avoid human error and bias influencing one’s decision-making, and thus avoiding the problems that affected act utilitarianism.
The error is that it’s humans who are attempting to implement the utilitarianism. I’m not talking about hypothetical non-human intelligences, and I don’t think they were implied in the context.
See also Ends Don’t Justify Means (Among Humans): having non-consequentialist rules (e.g. “Thou shalt not murder, even if it seems like a good idea”) can be consequentially desirable since we’re not capable of being ideal consequentialists.
Oh, indeed. But when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to.
I don’t think hypothetical superhuman would be dramatically different in their ability to employ predictive models upon uncertainty. If you increase power so it is to mankind as mankind is to 1 amoeba, you only double anything that is fundamentally logarithmic. While in many important cases there are faster approximations, it’s magical thinking to expect them everywhere; and there are problems where the errors inherently grow exponentially with time even if the model is magically perfect (butterfly effect). Plus, of course, models of other intelligences rapidly get unethical as you try to improve fidelity (if it is emulating people and putting them through torture and dust speck experience to compare values).
Well, those examples would have a lot of “okay we can’t calculate utility here, so we’ll use a principle” and far less faith in direct utilitarianism.
With the torture and dust specks, see, it arrives at counter intuitive conclusion, but it is not proof grade reasoning by any means. Who knows, maybe the correct algorithm for evaluation of torture vs dust specks must have BusyBeaver(10) for the torture, and BusyBeaver(9) for dust specks, or something equally outrageously huge (after all, thought, which is being screwed with by torture, is turing-complete). The 3^^^3 is not a very big number. There are numbers which are big like you wouldn’t believe.
edit: also I think even vastly superhuman entities wouldn’t be very good at consequence evaluation, especially from uncertain start state. In any case, some sorta morality oracle would have to be able to, at very least, take in full specs of human brain and then spit out the understanding of how to trade off the extreme pain of that individual, for dust speck of that individual (at task which may well end up in ultra long computations BusyBeaver(1E10) style. Forget the puny up arrow). That’s an enormously huge problem which the torture-choosers obviously a: haven’t done and b: didn’t even comprehend that something like this would be needed. Which brings us to the final point: the utilitarians are the people whom haven’t slightest clue what it might take to make an utilitarian decision, but are unaware of that deficiency. edit: and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence. Dust speck’s own value is still non comparable, it only bothers me because it creates the risk.
edit: note, the busy beaver reference is just an example. Before you can be additively operating on dust specks and pain, and start doing some utilitarian math there, you have to at least understand how the hell is it that an algorithm can be feeling pain, what is the pain exactly (in reductionist terms).
and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence
IIRC, in the original torture vs specks post EY specified that none of the dust specks would have any long-term consequence.
I know. Just wanted to point out where the personal preference (easily demonstrable when people e.g. neglect to take inconvenient safety measures) of small chance of torture vs definite dust speck comes from.
Yes. The error is that humans aren’t good at utilitarianism.
private_messaging has given an example elsewhere: the trouble with utilitarians is that they think they are utilitarians. They then use numbers to convince themselves to do something they would otherwise consider evil.
The Soviet Union was an attempt to build a Friendly government based on utilitarianism. They quickly reached “shoot someone versus dust specks” and went for shooting people.
They weren’t that good at lesser utilitarian decisions either, tending to ignore how humans actually behaved in favour of taking their theories and shutting-up-and-multiplying. Then when that didn’t work, they did it harder.
I’m sure someone objecting to the Soviet Union example as non-negligible evidence can come up with examples that worked out much better, of course.
See Eliezer’s Ethical Injunctions post.
Also Bryan Caplan:
As I have noted, when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to. If arithmetical utilitarianism works so well, it would work in weird territory.
Caplan does have a cultural point on the Soviet Union example. OTOH, it does seem a bit “no true utilitarian”.
Note the bank robbery thread below. Someone claims that “the utilitarian math” shows that robbing banks and donating to charity would have the best consequences. But they don’t do any math or look up basic statistics to do a Fermi calculation. A few minutes of effort shows that bank robbery actually pays much worse than working as a bank teller over the course of a career (including jail time, etc).
In Giving What We Can there are several people who donate half their income (or all income above a Western middle class standard of living) to highly efficient charities helping people in the developing world. They expect to donate millions of dollars over their careers, and to have large effects on others through their examples and reputations, both as individuals and via their impact on organizations like Giving What We Can. They do try to actually work things out, and basic calculations easily show that running around stealing organs or robbing banks would have terrible consequences, thanks to strong empirical regularities:
Crime mostly doesn’t pay. Bank robbers, drug dealers, and the like make less than legitimate careers. They also spend a big chunk of time imprisoned, and ruin their employability for the future. Very talented people who might do better than the average criminal can instead go to Wall Street or Silicon Valley and make far more.
Enormous amounts of good can be done through a normal legitimate career. Committing violent crime, or other hated acts close off such opportunities very rapidly.
Really dedicated do-gooders hope to have most of their influence through example, encouraging others to do good. Becoming a hated criminal, and associating their ethical views with such, should be expected to have huge negative effects by staunching the flow of do-gooders to exploit the vast legitimate opportunities to help people.
If some criminal scheme looks easy and low-risk, consider that law enforcement uses many techniques which are not made public, and are very hard for a lone individual to learn. There are honey-pots, confederates, and so forth. In the market for nuclear materials, most of the buyers and sellers are law enforcement agents trying to capture any real criminal participants. In North America terrorist cells are now regularly infiltrated long before they act, with government informants insinuated into the cell, phone and internet activities monitored, etc.
It is hard to keep a crime secret over time. People feel terrible guilt, and often are caught after they confess to others. In the medium term there is some chance of more effective neuroscience-based lie detectors, which goes still higher long-term.
The broader society, over time, could punish utilitarian villainy by reducing its support for the things utilitarians seek as they are associated with villains, or even by producing utilitarian evils. If animal rights terrorists tried to kill off humanity, it might lead to angry people eating more meat or creating anti-utilitronium (by the terrorists’ standards, not so much the broader society, focused on animals, say) in anger. The 9/11 attacks were not good for Osama bin Laden’s ambitions of ruling Saudi Arabia.
There are other considerations, but these are enough to dispense with the vast bestiary of supposedly utility-boosting sorts of wrongdoing. Arithmetical utilitarianism does say you should not try to become a crook. But unstable or vicious people (see the Caplan Leninist link) sometimes do like to take the idea of “the end justifies the means” as an excuse to go commit crimes without even trying to work out how the means are related to the end, and to alternatives.
Disclaimer: I do not value total welfare to the exclusion of other ethical and personal concerns. My moral feelings oppose deontological nastiness aside from aggregate welfare. But I am tired of straw-manning “estimating consequences” and “utilitarian math” by giving examples where these aren’t used and would have prevented the evil conclusion supposedly attributed to them.
I’m confused. Your comment paints a picture of a super-efficient police force that infiltrates criminal groups long before they act. But the Internet seems to say that many gangs in the US operate openly for years, control whole neighborhoods, and have their own Wikipedia pages...
The gangs do well, and the rare criminals who become successful gang leaders may sometimes do well, but does the average gangster do well?
Gang membership still doesn’t pay relative to regular jobs
The police largely know who is in the gangs, and can crack down if this becomes a higher priority
Terrorism is such a priority, to a degree way out of line with the average historical damage, because of 9/11; many have critiqued the diversion of law enforcement resources to terrorism
Such levels of gang control are concentrated in poor areas with less police funding, and areas where the police are estranged from the populace, limiting police activity.
Gang violence is heavily directed at other criminal gangs, reducing the enthusiasm of law enforcement, relative to more photogenic victims
The other side is that robbing banks at gunpoint isn’t the most effective way to redistribute wealth from those who have it to those to whom it should go.
I suspect that the most efficient way to do that is government seizure- declare that the privately held assets of the bank now belong to the charities. That doesn’t work, because the money isn’t value, it’s a signifier of value, and rewriting the map does not change the territory- if money is forcibly redistributed too much, it loses too much value and the only way to enforce the tax collection is by using the threat of prison and execution- but the jailors and executioners can only be paid by the taxes. Effectively robbing banks to give the money to charity harms everyone significantly, and fails to be better than doing nothing.
It may have been better if CarlShulman used a different word—perhaps ‘Evil’ - to represent the ‘ethical injunctions’ idea. That seems to better represent the whole “deliberately subvert consequentialist reasoning in certain areas due to acknowledgement of corrupted and bounded hardware”. ‘Weird’ seems to be exactly the sort of thing Eliezer might advocate. For example “make yourself into a corpsicle” and “donate to SingInst”.
But, of course, “weird” versus “evil” is not even broadly agreed upon.
And “weird” includes many things Eliezer advocates, but I would be very surprised if it did not include things that Eliezer most certainly would not advocate.
Of course it does. For example: dressing up as a penguin and beating people to death with a live fish. But that’s largely irrelevant. Rejecting ‘weird’ as the class of things that must never be done is not the same thing as saying that all things in that class must be done. Instead, weirdness is just ignored.
I’ve always felt that post was very suspect. Because, if you do the utilitarian math, robbing banks and giving them to charity is still a good deal, even if there’s a very low chance of it working. Your own welfare simply doesn’t play a factor, given the size of the variables you’re playing with. It seems to be that there is a deeper moral reason not to murder organ donors or steal food for the hungry than ‘it might end poorly for you.’
Bank robbery is actually unprofitable. Even setting aside reputation (personal and for one’s ethos), “what if others reasoned similarly,” the negative consequences of the robbery, and so forth you’d generate more expected income working an honest job. This isn’t a coincidence. Bank robbery hurts banks, insurers, and ultimately bank customers, and so they are willing to pay to make it unprofitable.
That was a somewhat lazy example, I admit, but consider the most inconvenient possible world. Let’s say you could expect to take a great deal more from a bank robbery. Would it then be valid utilitarian ethics to rob (indirectly) from the rich (us) to give to the poor?
My whole point in the comments on this post has been that it’s a pernicious practice to use such false examples. They leave erroneous impressions and associations. A world where bank-robbery is super-profitable, so profitable as to outweigh the effects of reputation and the like, is not very coherent.
A better example would be something like: “would utilitarians support raising taxes to fund malaria eradication,” or “would a utilitarian who somehow inherited swoopo.com (a dollar auction site) shut down the site or use the revenue to save kids from malaria” or “if a utilitarian inherited the throne in a monarchy like Oman (without the consent of the people) would he spend tax revenues on international good causes or return them to the taxpayers?”
Only if you’re bad at math. Banks aren’t just piggybanks to smash, they perform a useful function in the economy, and to disrupt it has consequences.
Of course I prefer to defeat bad utilitarian math with better utilitarian math rather than with ethical injunctions. But hey, that’s the woe of bounded reason, even without going into the whole corrupted hardware problem: your model is only so good, and heuristics that serve as warning signals have their place.
Why would that be an error? It’s not a requirement for an ethical theory that Homo sapiens must be good at it. If we notice that humans are bad at it, maybe we should make AI or posthumans that are better at it, if we truly view this as the best ethical theory. Besides, if the outcome of people following utilitarianism is really that bad, then utilitarianism would demand (it gets meta now) that people should follow some other theory that overall has better outcomes (see also Parfit’s Reasons and Persons). Another solution is Hare’s proposed “Two-Level Utilitarianism”. From Wikipedia:
The error is that it’s humans who are attempting to implement the utilitarianism. I’m not talking about hypothetical non-human intelligences, and I don’t think they were implied in the context.
See also Ends Don’t Justify Means (Among Humans): having non-consequentialist rules (e.g. “Thou shalt not murder, even if it seems like a good idea”) can be consequentially desirable since we’re not capable of being ideal consequentialists.
Oh, indeed. But when you’ve repeatedly emphasised “shut up and multiply”, tacking “btw don’t do anything weird” on the end strikes me as susceptible to your readers not heeding it, particularly when they really need to.
I don’t think hypothetical superhuman would be dramatically different in their ability to employ predictive models upon uncertainty. If you increase power so it is to mankind as mankind is to 1 amoeba, you only double anything that is fundamentally logarithmic. While in many important cases there are faster approximations, it’s magical thinking to expect them everywhere; and there are problems where the errors inherently grow exponentially with time even if the model is magically perfect (butterfly effect). Plus, of course, models of other intelligences rapidly get unethical as you try to improve fidelity (if it is emulating people and putting them through torture and dust speck experience to compare values).
Well, those examples would have a lot of “okay we can’t calculate utility here, so we’ll use a principle” and far less faith in direct utilitarianism.
With the torture and dust specks, see, it arrives at counter intuitive conclusion, but it is not proof grade reasoning by any means. Who knows, maybe the correct algorithm for evaluation of torture vs dust specks must have BusyBeaver(10) for the torture, and BusyBeaver(9) for dust specks, or something equally outrageously huge (after all, thought, which is being screwed with by torture, is turing-complete). The 3^^^3 is not a very big number. There are numbers which are big like you wouldn’t believe.
edit: also I think even vastly superhuman entities wouldn’t be very good at consequence evaluation, especially from uncertain start state. In any case, some sorta morality oracle would have to be able to, at very least, take in full specs of human brain and then spit out the understanding of how to trade off the extreme pain of that individual, for dust speck of that individual (at task which may well end up in ultra long computations BusyBeaver(1E10) style. Forget the puny up arrow). That’s an enormously huge problem which the torture-choosers obviously a: haven’t done and b: didn’t even comprehend that something like this would be needed. Which brings us to the final point: the utilitarians are the people whom haven’t slightest clue what it might take to make an utilitarian decision, but are unaware of that deficiency. edit: and also, I would likely take 1/3^^^3 chance of torture over a dust speck. Why? Because dust speck may result in an accident leading up to decades of torturous existence. Dust speck’s own value is still non comparable, it only bothers me because it creates the risk.
edit: note, the busy beaver reference is just an example. Before you can be additively operating on dust specks and pain, and start doing some utilitarian math there, you have to at least understand how the hell is it that an algorithm can be feeling pain, what is the pain exactly (in reductionist terms).
IIRC, in the original torture vs specks post EY specified that none of the dust specks would have any long-term consequence.
I know. Just wanted to point out where the personal preference (easily demonstrable when people e.g. neglect to take inconvenient safety measures) of small chance of torture vs definite dust speck comes from.