First, “caring about qualia” is meant to be a very weak statement, like “caring at all, all possible ways of caring”, not “maximize qualia”. Second, this is a toy example, meant to convey the shape of how certain sort of training process can break when trained system becomes smarter, not overarching claim about correct morality. Why are you nitpicking toy example.
I’m afraid I have a habit, when someone makes what sounds like an AI alignment target proposal that I believe to be existentially risky, of pointing this fact out — if only to any readers who might otherwise be nodding along and thinking “that sounds very reasonable, no one could object to training AI to think that…”. I completely agree that I was assuming several steps between “cares about qualia” and “qualia maximizer” — steps that are admittedly common on LessWrong, but that you may well not have intended. Please take this in the spirit of a public service announcement of existential danger on the subject of this particular ethical system as an alignment target for AI, not a criticism of your ideas or of the use of this ethical viewpoint by a human. Re-reading you more carefully, you were actually describing an ex-Christian human with this viewpoint, and then analogizing an AI to that person, so it wasn’t actually clear whether you were proposing this as an ethical belief that we should aim to align AI to, or not — possibly you weren’t, in which case my nitpicking was unnecessary.
First, “caring about qualia” is meant to be a very weak statement, like “caring at all, all possible ways of caring”, not “maximize qualia”. Second, this is a toy example, meant to convey the shape of how certain sort of training process can break when trained system becomes smarter, not overarching claim about correct morality. Why are you nitpicking toy example.
I’m afraid I have a habit, when someone makes what sounds like an AI alignment target proposal that I believe to be existentially risky, of pointing this fact out — if only to any readers who might otherwise be nodding along and thinking “that sounds very reasonable, no one could object to training AI to think that…”. I completely agree that I was assuming several steps between “cares about qualia” and “qualia maximizer” — steps that are admittedly common on LessWrong, but that you may well not have intended. Please take this in the spirit of a public service announcement of existential danger on the subject of this particular ethical system as an alignment target for AI, not a criticism of your ideas or of the use of this ethical viewpoint by a human. Re-reading you more carefully, you were actually describing an ex-Christian human with this viewpoint, and then analogizing an AI to that person, so it wasn’t actually clear whether you were proposing this as an ethical belief that we should aim to align AI to, or not — possibly you weren’t, in which case my nitpicking was unnecessary.