I understand why the notions exist—I was trying to address the question of ‘what explainable-moral-intuitions should we keep as terminal values, and how do we tell them apart from those we shouldn’t’.
But your first sentence is taken very much to heart, sir.
Maybe I’m being silly here, in hindsight. Certain intuitive desires are reducible to others, and some, like ‘love/happiness/fun/etc.’ are probably not. It feels obvious that most people should immediately see that. Yes, they want a given ethical injunction to be obeyed, but not as a fundamental/terminal value.
Then again—there are Catholic moralists, including, I think, some Catholics I know personally, who firmly believe that (for example) stealing is wrong because stealing is wrong. Not for any other reason. Not because it brings harm to the person being stolen from. If you bring up exceptions—‘what about an orphan who will starve if they don’t steal that bread?’ they argue that this doesn’t count as stealing, not that it ‘proves that stealing isn’t really wrong.’ For them, every exception is simply to be included as another fundamental rule. At least, that’s the mindset, as far as I can tell. I saw the specific argument above being formulated for use against moral relativists, who were apparently out to destroy society by showing that different things were right for different people.
Even though this article is about AI, and even though we should not trust ourselves to understand when we should be excepted from an injunction—this seems like a belief that might eventually have some negative real-world consequences. See, potentially, ‘homosexuality is wrong because homosexuality is wrong’?
If I tried to tell any of these people about how ethical injunctions could be explained as heuristics for achieving higher terminal values—I can already feel myself being accused of shuffling things around, trying to convert goods into other incompatible goods in order to justify some sinister, contradictory worldview.
If I brought up reductionism, it seems almost trivial—while I’m simulating their mind—to point out that no one has ever provably applied reduction to morals.
So maybe let me rephrase: is there any way I could talk them out of it?
I think this answer contains something important--
Not so much an answer to the problem, but a clue to the reason WHY we intuitively, as humans, know to respond in a way which seems un-mathematical.
It seems like a Game Theory problem to me. Here, we’re calling the opponents’ bluff. If we make the decision that SEEMINGLY MAXIMIZES OUR UTILITY, according to game theory we’re set up for a world of hurt in terms of indefinite situations where we can be taken advantage of. Game Theory already contains lots of situations where reasons exist to take action that seemingly does not maximize your own utility.