This makes me think that a useful meta-principle for application of moral principles in the absence omniscience is “robustness to auxillary information.” Phrased another way, if the variance of the outcomes of your choices is high according to a moral principle, in all but the most extreme cases, either find more information or pick a different moral principle.
Can you maybe give an example or two (or one example and one counter example) to help illustrate how a moral principle displaying “robustness to auxiliary information” operates in practice, versus one that does not? Specifically, I’m interested in understanding how the variance in outcomes might manifest with the addition of new information.
Let’s consider the trolley problem. One consequentialist solution is “whichever choice leads to the best utility over the lifetime of the universe”, which is intractable. This meta-principle rules it out as follows: if, for example, you learned that one of the 5 was on the brink of starting a nuclear war and the lone one was on the brink of curing aging, that would say switch, but if the two identities were flipped, it would say stay, and generally, there are too many unobservables to consider. By contrast, a simple utilitarian approach of “always switch” is allowed by the principle, as are approaches that take into account demographics or personal importance.
The principle also suggests that killing a random person on the street is bad, even if the person turns out to be plotting a mass murder, and conversely, a doctor saving said person’s life is good.
Two additional cases where the principle may be useful and doesn’t completely correspond to common sense:
I once read an article by a former vegan arguing against veganism and vegetarianism; one example was the fact that the act of harvesting grain involves many painful deaths of field mice, and that’s not particularly better than killing one cow. Applying the principle, this suggests that suffering or indirect death cannot straightforwardly be the basis for these dietary choices, and that consent is on shaky ground.
When thinking about building a tool (like the LW infrastructure) that could be either hugely positive (because it leads to aligned AI) or hugely negative (because it leads to unaligned AI by increasing AI discussions), and there isn’t really a way to know which, you are morally free to build it or not; any steps you take to increase the likelihood of a positive outcome are good, but you are not required to stop building the tool due to a huge unknowable risk. Of course, if there’s compelling reason to believe that the tool is net-negative, that reduces the variance and suggests that you shouldn’t build it (e.g. most AI capabilities advancements).
Framed a different way, the principle is, “Don’t tie yourself up in knots overthinking.” It’s slightly reminiscent of quiescence search in that it’s solving a similar “horizon effect” problem, but it’s doing so by discarding evaluation heuristics that are not locally stable.
Thanks for the explanation. This makes a lot of sense to me now. I’m glad I asked!
While I agree that there is value in “don’t tie yourself up in knots overthinking”, my intuition tells me that there is a lot of value in just knowing about / considering that there is more information about a situation to be had, which might, in theory, influence my decision about that situation in important ways. It changes how I engange with all kinds of situations beforehand, and also after the fact. So considering the motivations and backstories of the people in the trolley-problem does have value, even if in that particular moment I do not have the time to gather more information and a decision needs to be made quickly.
I don’t think that this point needs to be made for people on this forum. It’s more aimed at people who are looking for rules / strategies / heuristics to robotically and mindlessly apply them to their lives (and enforce those rules for others).
This makes me think that a useful meta-principle for application of moral principles in the absence omniscience is “robustness to auxillary information.” Phrased another way, if the variance of the outcomes of your choices is high according to a moral principle, in all but the most extreme cases, either find more information or pick a different moral principle.
This sounds intuitively interesting to me.
Can you maybe give an example or two (or one example and one counter example) to help illustrate how a moral principle displaying “robustness to auxiliary information” operates in practice, versus one that does not? Specifically, I’m interested in understanding how the variance in outcomes might manifest with the addition of new information.
Let’s consider the trolley problem. One consequentialist solution is “whichever choice leads to the best utility over the lifetime of the universe”, which is intractable. This meta-principle rules it out as follows: if, for example, you learned that one of the 5 was on the brink of starting a nuclear war and the lone one was on the brink of curing aging, that would say switch, but if the two identities were flipped, it would say stay, and generally, there are too many unobservables to consider. By contrast, a simple utilitarian approach of “always switch” is allowed by the principle, as are approaches that take into account demographics or personal importance.
The principle also suggests that killing a random person on the street is bad, even if the person turns out to be plotting a mass murder, and conversely, a doctor saving said person’s life is good.
Two additional cases where the principle may be useful and doesn’t completely correspond to common sense:
I once read an article by a former vegan arguing against veganism and vegetarianism; one example was the fact that the act of harvesting grain involves many painful deaths of field mice, and that’s not particularly better than killing one cow. Applying the principle, this suggests that suffering or indirect death cannot straightforwardly be the basis for these dietary choices, and that consent is on shaky ground.
When thinking about building a tool (like the LW infrastructure) that could be either hugely positive (because it leads to aligned AI) or hugely negative (because it leads to unaligned AI by increasing AI discussions), and there isn’t really a way to know which, you are morally free to build it or not; any steps you take to increase the likelihood of a positive outcome are good, but you are not required to stop building the tool due to a huge unknowable risk. Of course, if there’s compelling reason to believe that the tool is net-negative, that reduces the variance and suggests that you shouldn’t build it (e.g. most AI capabilities advancements).
Framed a different way, the principle is, “Don’t tie yourself up in knots overthinking.” It’s slightly reminiscent of quiescence search in that it’s solving a similar “horizon effect” problem, but it’s doing so by discarding evaluation heuristics that are not locally stable.
Thanks for the explanation. This makes a lot of sense to me now. I’m glad I asked!
While I agree that there is value in “don’t tie yourself up in knots overthinking”, my intuition tells me that there is a lot of value in just knowing about / considering that there is more information about a situation to be had, which might, in theory, influence my decision about that situation in important ways. It changes how I engange with all kinds of situations beforehand, and also after the fact. So considering the motivations and backstories of the people in the trolley-problem does have value, even if in that particular moment I do not have the time to gather more information and a decision needs to be made quickly.
I don’t think that this point needs to be made for people on this forum. It’s more aimed at people who are looking for rules / strategies / heuristics to robotically and mindlessly apply them to their lives (and enforce those rules for others).