because goodness is caring about beings with qualia, even if those beings are not Homo Sapiens.
If we are trying to align our ASI to the flourishing of humans, to make it an intelligent part of our extended phenotype, then this (philosophically popular) statement is unwise. Or, more accurately, it is true only in an inappropriate choice of ethical system.
What do I mean by “inappropriate”? That it’s an existentially dangerous choice of ethical system to align ASi to, for the humans training the ASI. To give a concrete example, ants almost certainly have qualia (or if you don’t believe that, consider mice instead, and adjust the following numbers). A quality-adjusted-life-year for an ant costs approximately one ten-millionth of the resources that a quality-adjusted-life-year for a human costs (and ants objectively live and think faster, presumably having more qualia-per-second, so the ratio may actually be even better). So an ASI aligned to the definition of goodness that you proposed would be very keen to replace the O(10 billion) humans on Earth with O(100 quadrillion) ants — or possibly even more of some even smaller organism. That is not human-aligned behavior, that is a qualia maximizer.
You might suggest that ants have less qualia, or less good qualia. Perhaps even less good by about a factor of ten million. Unless it happens that the qualia quality of every single species is exactly proportional of its resource cost for the currently available bundle of natural resources, which would seem an astonishing coincidence across tens of millions of species, the ethical instability remains. See my post Moral Value for Sentient Animals? Alas, Not Yet from my AI, Alignment, and Ethics sequence for a more detailed exposition.
First, “caring about qualia” is meant to be a very weak statement, like “caring at all, all possible ways of caring”, not “maximize qualia”. Second, this is a toy example, meant to convey the shape of how certain sort of training process can break when trained system becomes smarter, not overarching claim about correct morality. Why are you nitpicking toy example.
I’m afraid I have a habit, when someone makes what sounds like an AI alignment target proposal that I believe to be existentially risky, of pointing this fact out — if only to any readers who might otherwise be nodding along and thinking “that sounds very reasonable, no one could object to training AI to think that…”. I completely agree that I was assuming several steps between “cares about qualia” and “qualia maximizer” — steps that are admittedly common on LessWrong, but that you may well not have intended. Please take this in the spirit of a public service announcement of existential danger on the subject of this particular ethical system as an alignment target for AI, not a criticism of your ideas or of the use of this ethical viewpoint by a human. Re-reading you more carefully, you were actually describing an ex-Christian human with this viewpoint, and then analogizing an AI to that person, so it wasn’t actually clear whether you were proposing this as an ethical belief that we should aim to align AI to, or not — possibly you weren’t, in which case my nitpicking was unnecessary.
If we are trying to align our ASI to the flourishing of humans, to make it an intelligent part of our extended phenotype, then this (philosophically popular) statement is unwise. Or, more accurately, it is true only in an inappropriate choice of ethical system.
What do I mean by “inappropriate”? That it’s an existentially dangerous choice of ethical system to align ASi to, for the humans training the ASI. To give a concrete example, ants almost certainly have qualia (or if you don’t believe that, consider mice instead, and adjust the following numbers). A quality-adjusted-life-year for an ant costs approximately one ten-millionth of the resources that a quality-adjusted-life-year for a human costs (and ants objectively live and think faster, presumably having more qualia-per-second, so the ratio may actually be even better). So an ASI aligned to the definition of goodness that you proposed would be very keen to replace the O(10 billion) humans on Earth with O(100 quadrillion) ants — or possibly even more of some even smaller organism. That is not human-aligned behavior, that is a qualia maximizer.
You might suggest that ants have less qualia, or less good qualia. Perhaps even less good by about a factor of ten million. Unless it happens that the qualia quality of every single species is exactly proportional of its resource cost for the currently available bundle of natural resources, which would seem an astonishing coincidence across tens of millions of species, the ethical instability remains. See my post Moral Value for Sentient Animals? Alas, Not Yet from my AI, Alignment, and Ethics sequence for a more detailed exposition.
First, “caring about qualia” is meant to be a very weak statement, like “caring at all, all possible ways of caring”, not “maximize qualia”. Second, this is a toy example, meant to convey the shape of how certain sort of training process can break when trained system becomes smarter, not overarching claim about correct morality. Why are you nitpicking toy example.
I’m afraid I have a habit, when someone makes what sounds like an AI alignment target proposal that I believe to be existentially risky, of pointing this fact out — if only to any readers who might otherwise be nodding along and thinking “that sounds very reasonable, no one could object to training AI to think that…”. I completely agree that I was assuming several steps between “cares about qualia” and “qualia maximizer” — steps that are admittedly common on LessWrong, but that you may well not have intended. Please take this in the spirit of a public service announcement of existential danger on the subject of this particular ethical system as an alignment target for AI, not a criticism of your ideas or of the use of this ethical viewpoint by a human. Re-reading you more carefully, you were actually describing an ex-Christian human with this viewpoint, and then analogizing an AI to that person, so it wasn’t actually clear whether you were proposing this as an ethical belief that we should aim to align AI to, or not — possibly you weren’t, in which case my nitpicking was unnecessary.