Moral realism is probably false which kinda sucks. But we can probably replace it with some kind of souped-up moral subjectivism. That is, rather than discovering and optimising Goodness, we can instead discover and optimise something like “What would I ideally want?” for some appropriate idealisation procedure.
What if that doesn’t work? More generally, what if the notion of Goodness is deeply broken such that we can’t replace it with anything else? That is, there is nothing Goodness-shaped in the world, either objectively or subjectively. There is no adequate notion of choiceworthiness. Yuck.
What would we have left?
I think there are other ideals we could strive for, like Truth or Beauty. We currently strive for these ideals partly for moral reasons — and we would lose that additional oomph — but they are still decent ideals to fall back on by themselves.
I suspect Truth might fall as well (see reply), so we might be left with just Beauty, Fun, and maybe some other ideals which are more unmediated than Goodness or Truth. But I’d feel a bit shortchanged. I’m White/Blue in the MGT Colour Wheel, so Goodness and Truth are what I care most about.
I don’t think I would substantially regret my choices on Deep Nihilism. I think there are decent nihilistic justifications for working on AI safety (e.g. it’s fun, it’s cool, it makes me feel important, etc). And there are probably good nihilistic justifications at the organisational and societal levels as well, not just the individualist level.
Of course, there are some ways that I could’ve lived more nihilistically, e.g. been more “chill”, followed a broader range of intellectual pursuits. But these are pretty minor, I’ve made some sacrifices in my life for Truth or Goodness, but not big ones. My guess is this is due to “moral luck”: I enjoy interacting with people in the AI safety community, and I don’t enjoy interacting with people in policy/advocacy, and “luckily” I’m not fit for policy/advocacy. Maybe this is motivated reasoning / weaponised incompetence, and actually I would be great at policy/advocacy but I don’t want to make the sacrifice. Not sure.
Overall, I don’t feel that stressed about Deep Nihilism. I think (1) Deep Nihilism is pretty unlikely (<10%), (2) Deep Nihilism can be safely bracketed, (3) Deep Nihilism wouldn’t make me regret my actions substantially.
I think there are decent nihilistic justifications for working on AI safety (e.g. it’s fun, it’s cool, it makes me feel important, etc).
I think you are misunderstanding the implications of nihilism. Copying from here:
Compare “I want the oppressed masses to find justice” with “I’ve been standing too long, I want to sit down”. These two “wants” are fundamentally built out of the same mind-stuff. They both derive from positive valence, which in turn ultimately comes from innate drives (specifically, mainly social drives in the first case, and homeostatic energy-conserving drives in the second case). So if “true morality” or “true human morality” or whatever doesn’t exist, then that does not constitute a reason to sit down rather than to seek justice. You still have to make decisions. That’s what I meant by “nihilism is not decision-relevant”, or Yudkowsky by “What would you do without morality?”. …
However, nihilism is not decision-relevant. Imagine being a nihilist, deciding whether to spend your free time trying to bring about an awesome post-AGI utopia, vs sitting on the couch and watching TV. Well, if you’re a nihilist, then the awesome post-AGI utopia doesn’t matter. But watching TV doesn’t matter either. Watching TV entails less exertion of effort. But that doesn’t matter either. Watching TV is more fun (umm, for some people). But having fun doesn’t matter either. There’s no reason to throw yourself at a difficult project. There’s no reason NOT to throw yourself at a difficult project! So nihilism is just not a helpful decision criterion!! What else is there?
I propose a different starting point—what I call Dentin’s prayer: Why do I exist? Because the universe happens to be set up this way. Why do I care (about anything or everything)? Simply because my genetics, atoms, molecules, and processing architecture are set up in a way that happens to care. …
What would you call the thing where it turns out that all my morality-flavoured wants are nonsense, but all my selfish-flavoured wants still make sense? Can we sub that term into the OP in place of ‘nihilism’?
Or do you deny the premise that there are different flavours? I personally really feel like I can taste the flavours.
Maybe I’m missing the point, but I don’t get how Deep Nihilism could possibly be true. I don’t expect that there’s some neat moral theory that explains all my preferences and which I can safely optimize against, but there is some common-sense notion of the Good that makes me think “loving my family is better than murdering them.” If some idealization process causes me to reverse that preference… it’s probably the wrong idealization process.
As promised, here’s a plausible sketch for how Truth might fall:
Modal realism is true, so you lose all objective contingent truths (e.g. the sky is blue). You only have the truths about:
the space of all possible worlds (e.g. there is some world where the sky is blue, if the sky is possibly a colour, and the sea is possibly a colour, then it’s possible that the sky is the first colour and the sea is the second colour.)
subjective truths about which world you occupy (e.g. the sky in my world is blue). This is a bit more impoverished but probably fine.
However, you might lose the subjective contingent truths also, because the “I/me/my” might pick out multiple entities in different possible worlds, e.g. some entities in base reality, some in a simulated reality, etc. That kinda sucks, because then there is not even a subjective truth about whether you’re in a simulation. Note this is worse than the sceptical “you don’t know if you’re in a simulation” — instead the worry is that there’s no truth to know!
Possibly you can save this with UDASSA-ish, epistemology is caring. But then we’re at square zero. Your “caring” might not be coherent even under idealisation.
Okay so we’re left with what? Just the facts about different Solomonoff priors? That’s kinda impoverished. And we might even lose those facts due to logically impossible worlds, or non-standard arithmetic or set-theoretic pluralism.
I don’t worry about this, but I do worry about worlds where objectiveMorality_1 (goal convergence through idealized reflective reasoning) diverges significantly from objectiveMorality_2 (what everyone would prefer everyone else to value, or rules everyone would want to bind everyone). In this case we can bully or brainwash each other into a cooperative equilibrium, but it would be a reasoning error (or at best lucky coincidence) not to defect.
Claude’s constitution says (I forget the exact phrasing) that it should do what is objectively moral if there is objective morality, and pursue everyone’s collective interest even if there is not. That’s good (objectively, or “just” for everybody) if OM1 = OM2 or if OM1 = null, but not otherwise.
My best guess is that these do align via galaxy brained Kantian decision-theoretic reasons, that are not actually so galaxy brained but show up in convergent formulations of the golden rule, but this could all be wishful thinking.
I don’t understand. Nothing bridges the is-ought divide. The notion of “goodness” is purely subjective and socially constructed.
But everything interesting about humanity is subjective and socially constructed. That doesn’t make it meaningless, it just reinforces that meaning is personal and subjective, and includes others because we care about others, in a recursive-reinforcement way.
Why is altruism or AI Safety only important if there’s a god or physical law behind it? People are important (to me, to you, to other people). Help them!
See the first paragraph — I think moral subjectivism is a bit less attractive than Objective Morality, as an ideal, but not much worse. By “Deep Nihilism” I mean that there isn’t even a coherent subjective morality.
What if Deep Nihilism is true?
Moral realism is probably false which kinda sucks. But we can probably replace it with some kind of souped-up moral subjectivism. That is, rather than discovering and optimising Goodness, we can instead discover and optimise something like “What would I ideally want?” for some appropriate idealisation procedure.
What if that doesn’t work? More generally, what if the notion of Goodness is deeply broken such that we can’t replace it with anything else? That is, there is nothing Goodness-shaped in the world, either objectively or subjectively. There is no adequate notion of choiceworthiness. Yuck.
What would we have left?
I think there are other ideals we could strive for, like Truth or Beauty. We currently strive for these ideals partly for moral reasons — and we would lose that additional oomph — but they are still decent ideals to fall back on by themselves.
I suspect Truth might fall as well (see reply), so we might be left with just Beauty, Fun, and maybe some other ideals which are more unmediated than Goodness or Truth. But I’d feel a bit shortchanged. I’m White/Blue in the MGT Colour Wheel, so Goodness and Truth are what I care most about.
I don’t think I would substantially regret my choices on Deep Nihilism. I think there are decent nihilistic justifications for working on AI safety (e.g. it’s fun, it’s cool, it makes me feel important, etc). And there are probably good nihilistic justifications at the organisational and societal levels as well, not just the individualist level.
Of course, there are some ways that I could’ve lived more nihilistically, e.g. been more “chill”, followed a broader range of intellectual pursuits. But these are pretty minor, I’ve made some sacrifices in my life for Truth or Goodness, but not big ones. My guess is this is due to “moral luck”: I enjoy interacting with people in the AI safety community, and I don’t enjoy interacting with people in policy/advocacy, and “luckily” I’m not fit for policy/advocacy. Maybe this is motivated reasoning / weaponised incompetence, and actually I would be great at policy/advocacy but I don’t want to make the sacrifice. Not sure.
Overall, I don’t feel that stressed about Deep Nihilism. I think (1) Deep Nihilism is pretty unlikely (<10%), (2) Deep Nihilism can be safely bracketed, (3) Deep Nihilism wouldn’t make me regret my actions substantially.
I think you are misunderstanding the implications of nihilism. Copying from here:
Here’s that link in the last sentence:
What would you call the thing where it turns out that all my morality-flavoured wants are nonsense, but all my selfish-flavoured wants still make sense? Can we sub that term into the OP in place of ‘nihilism’?
Or do you deny the premise that there are different flavours? I personally really feel like I can taste the flavours.
The link is currently broken, this appears to be from [Valence series] 2. Valence & Normativity
Maybe I’m missing the point, but I don’t get how Deep Nihilism could possibly be true. I don’t expect that there’s some neat moral theory that explains all my preferences and which I can safely optimize against, but there is some common-sense notion of the Good that makes me think “loving my family is better than murdering them.” If some idealization process causes me to reverse that preference… it’s probably the wrong idealization process.
As promised, here’s a plausible sketch for how Truth might fall:
Modal realism is true, so you lose all objective contingent truths (e.g. the sky is blue). You only have the truths about:
the space of all possible worlds (e.g. there is some world where the sky is blue, if the sky is possibly a colour, and the sea is possibly a colour, then it’s possible that the sky is the first colour and the sea is the second colour.)
subjective truths about which world you occupy (e.g. the sky in my world is blue). This is a bit more impoverished but probably fine.
However, you might lose the subjective contingent truths also, because the “I/me/my” might pick out multiple entities in different possible worlds, e.g. some entities in base reality, some in a simulated reality, etc. That kinda sucks, because then there is not even a subjective truth about whether you’re in a simulation. Note this is worse than the sceptical “you don’t know if you’re in a simulation” — instead the worry is that there’s no truth to know!
Possibly you can save this with UDASSA-ish, epistemology is caring. But then we’re at square zero. Your “caring” might not be coherent even under idealisation.
Okay so we’re left with what? Just the facts about different Solomonoff priors? That’s kinda impoverished. And we might even lose those facts due to logically impossible worlds, or non-standard arithmetic or set-theoretic pluralism.
I don’t worry about this, but I do worry about worlds where objectiveMorality_1 (goal convergence through idealized reflective reasoning) diverges significantly from objectiveMorality_2 (what everyone would prefer everyone else to value, or rules everyone would want to bind everyone). In this case we can bully or brainwash each other into a cooperative equilibrium, but it would be a reasoning error (or at best lucky coincidence) not to defect.
Claude’s constitution says (I forget the exact phrasing) that it should do what is objectively moral if there is objective morality, and pursue everyone’s collective interest even if there is not. That’s good (objectively, or “just” for everybody) if OM1 = OM2 or if OM1 = null, but not otherwise.
My best guess is that these do align via galaxy brained Kantian decision-theoretic reasons, that are not actually so galaxy brained but show up in convergent formulations of the golden rule, but this could all be wishful thinking.
what do you mean by Deep Nihilism
There is no sufficiently coherent/nice/appropriate morality, either objective or subjective, even after idealisation.
I don’t understand. Nothing bridges the is-ought divide. The notion of “goodness” is purely subjective and socially constructed.
But everything interesting about humanity is subjective and socially constructed. That doesn’t make it meaningless, it just reinforces that meaning is personal and subjective, and includes others because we care about others, in a recursive-reinforcement way.
Why is altruism or AI Safety only important if there’s a god or physical law behind it? People are important (to me, to you, to other people). Help them!
See the first paragraph — I think moral subjectivism is a bit less attractive than Objective Morality, as an ideal, but not much worse. By “Deep Nihilism” I mean that there isn’t even a coherent subjective morality.