Well, if a different thought experiment leads to different ‘folk-morality’-based conclusions while it doesn’t make a difference from a strictly utilitarian view point, that shows they are not fully compatible, or? Obviously, you can make them agree again by adding things, but that does not resolve the original problem.
For the success of an ethical system indeed it’s important to resonate with folk morality, but I also think the phenotype of everyday folk morality is a hodge-podge of biases and illusions. If we would take the basics of folk morality (what would that be, maybe… golden rule + utilitarianism?) I think something more consistent could be forged.
I have two responses, based on two different interpretations of your response:
I don’t see why a “strict utilitarian” would have to be a first-order utilitarian, i.e. one that only sees the immediate consequences of its actions and not the consequences of others’ responses to its actions. To avoid dealing with the social consequences (that is, extrapolated multi-actor consequences) of an act means to imagine it being performed in a moral vacuum: a place where nobody outside the imagined scene has any way of finding out what happened or responding to it. But breaking a rule when nobody is around to be injured by it or to report your rule-breaking to others, is a significantly different thought-experiment from breaking a rule under normal circumstances.
Humans (and, perhaps, AIs) are not designed to live in a world where the self is the only morally significant actor. They have to care about what other morally significant persons care about, and at more than a second-order level: they need to see not just smiling faces, but have the knowledge that there are minds behind those smiling faces.
And in any event, human cognition does not perform well in social or cognitive isolation: put a person in solitary confinement and you can predict that he or she will suffer and experience certain forms of cognitive breakdown; keep a scientific mind in isolation from a scientific community and you can expect that you will get a kook, not a genius.
Some people seem to treat the trolley problem and the doctor problem as the same problem stated in two different ways, in such a way as to expose a discrepancy in human moral reasoning. If so, this might be analogous to the Wason selection task, which exposes a discrepancy in human symbolic reasoning. Wason demonstrates that humans reason more effectively about applying social rules than about applying arithmetical rules isomorphic to those social rules. I’ve always imagined this as humans using a “social coprocessor” to evaluate rule-following, which is not engaged when thinking about an arithmetical problem.
Perhaps something similar is going on in the trolley problem and doctor problem: we are engaging a different “moral coprocessor” to think about one than the other. This difference may be captured by different schools of ethics: the doctor problem engages a “deontological coprocessor”, wherein facts such as a doctor’s social role and expected consequences of betrayal of duty are relevant. The trolley problem falls back to the “consequentialist coprocessor” for most readers, though, which computes: five dead is worse than one dead.
Perhaps the consequentialist coprocessor is a stupid, first-order utilitarian, whereas the deontological coprocessor deals with the limit case of others’ responses to our acts better.
Well, if a different thought experiment leads to different ‘folk-morality’-based conclusions while it doesn’t make a difference from a strictly utilitarian view point, that shows they are not fully compatible, or? Obviously, you can make them agree again by adding things, but that does not resolve the original problem.
For the success of an ethical system indeed it’s important to resonate with folk morality, but I also think the phenotype of everyday folk morality is a hodge-podge of biases and illusions. If we would take the basics of folk morality (what would that be, maybe… golden rule + utilitarianism?) I think something more consistent could be forged.
I have two responses, based on two different interpretations of your response:
I don’t see why a “strict utilitarian” would have to be a first-order utilitarian, i.e. one that only sees the immediate consequences of its actions and not the consequences of others’ responses to its actions. To avoid dealing with the social consequences (that is, extrapolated multi-actor consequences) of an act means to imagine it being performed in a moral vacuum: a place where nobody outside the imagined scene has any way of finding out what happened or responding to it. But breaking a rule when nobody is around to be injured by it or to report your rule-breaking to others, is a significantly different thought-experiment from breaking a rule under normal circumstances.
Humans (and, perhaps, AIs) are not designed to live in a world where the self is the only morally significant actor. They have to care about what other morally significant persons care about, and at more than a second-order level: they need to see not just smiling faces, but have the knowledge that there are minds behind those smiling faces.
And in any event, human cognition does not perform well in social or cognitive isolation: put a person in solitary confinement and you can predict that he or she will suffer and experience certain forms of cognitive breakdown; keep a scientific mind in isolation from a scientific community and you can expect that you will get a kook, not a genius.
Some people seem to treat the trolley problem and the doctor problem as the same problem stated in two different ways, in such a way as to expose a discrepancy in human moral reasoning. If so, this might be analogous to the Wason selection task, which exposes a discrepancy in human symbolic reasoning. Wason demonstrates that humans reason more effectively about applying social rules than about applying arithmetical rules isomorphic to those social rules. I’ve always imagined this as humans using a “social coprocessor” to evaluate rule-following, which is not engaged when thinking about an arithmetical problem.
Perhaps something similar is going on in the trolley problem and doctor problem: we are engaging a different “moral coprocessor” to think about one than the other. This difference may be captured by different schools of ethics: the doctor problem engages a “deontological coprocessor”, wherein facts such as a doctor’s social role and expected consequences of betrayal of duty are relevant. The trolley problem falls back to the “consequentialist coprocessor” for most readers, though, which computes: five dead is worse than one dead.
Perhaps the consequentialist coprocessor is a stupid, first-order utilitarian, whereas the deontological coprocessor deals with the limit case of others’ responses to our acts better.