I have what feels like a naive question. Is there any reason that we can’t keep appealing to even higher-order preferences? I mean, when I find that I have these sorts of inconsistencies, I find myself making an additional moral judgment that tries to resolve the inconsistency. So couldn’t you show the human (or, if the AI is doing all this in its ‘head’, a suitably accurate simulation of the human) that their preference depends on the philosopher that we introduce them to? Or in other cases where, say, ordering matters, show them multiple orderings, or their simulations’ reactions to every possible ordering where feasible, and so on. Maybe this will elicit a new judgment that we would consider morally relevant. But this all relies on simulation, I don’t know if you can get the same effect without that capability, and this solution doesn’t seem even close to being fully general.
I imagine that this might not do much to resolve your confusion however. It doesn’t do much to resolve mine.
It seems to me to jive with how many people react to unexpected tensions between different parts of their values (eg Global warming vs markets solve everything, or Global warming vs nuclear power is bad). If the tension can’t be ignored or justified away, they often seem to base their new decision on affect and social factors, far more than any principled meta-preference for how to resolve tensions.
But you can still keep asking the “why” question and go back dozens of layers, usually suffering combinatorial explosion of causes, and even recursion in some cases. Only very, very rarely have I ever encountered a terminal, genesis cause for which there isn’t a “why”—the will to live is honestly the only one occurring to me right now. Everything else has causes upon causes as far as I’d care to look...
I have what feels like a naive question. Is there any reason that we can’t keep appealing to even higher-order preferences? I mean, when I find that I have these sorts of inconsistencies, I find myself making an additional moral judgment that tries to resolve the inconsistency. So couldn’t you show the human (or, if the AI is doing all this in its ‘head’, a suitably accurate simulation of the human) that their preference depends on the philosopher that we introduce them to? Or in other cases where, say, ordering matters, show them multiple orderings, or their simulations’ reactions to every possible ordering where feasible, and so on. Maybe this will elicit a new judgment that we would consider morally relevant. But this all relies on simulation, I don’t know if you can get the same effect without that capability, and this solution doesn’t seem even close to being fully general.
I imagine that this might not do much to resolve your confusion however. It doesn’t do much to resolve mine.
I don’t think most humans have higher order preferences, beyond, say, two levels max.
Okay well that doesn’t jive with my own introspective experience.
It seems to me to jive with how many people react to unexpected tensions between different parts of their values (eg Global warming vs markets solve everything, or Global warming vs nuclear power is bad). If the tension can’t be ignored or justified away, they often seem to base their new decision on affect and social factors, far more than any principled meta-preference for how to resolve tensions.
But you can still keep asking the “why” question and go back dozens of layers, usually suffering combinatorial explosion of causes, and even recursion in some cases. Only very, very rarely have I ever encountered a terminal, genesis cause for which there isn’t a “why”—the will to live is honestly the only one occurring to me right now. Everything else has causes upon causes as far as I’d care to look...
Oh, in that sense, yeah. I meant as in having articulated meta-preferences that explain lower level preferences.