You’re correct that the essay is backchainy. Stratified utopia is my current best bet for “most desirable future given our moral uncertainty” which motivates me to evaluate its likelihood. I don’t think it’s very likely, maybe 5-10%, and I could easily shift with further thought
Starting with the most desirable future and then evaluating its likelihood does risk privileging the hypothesis. This is a fair critique: better epistemics would start with the most likely future and then evaluate its desirability.
(2)
Regarding your example: “if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise.”
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
More common are cases involving contested sacred sites, which also violate the Nothing-To-Mourn Principle. For example, some people would mourn if the Third Temple were never rebuilt on the Temple Mount, while others would mourn if the Al-Aqsa Mosque were destroyed to make way for it.
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
I agree that this is an empirical claim, in fact, this is pretty much the major interesting question here! Please say more about why you think this is the case. Empirically, Brian Tomasik does exist, as do nature-has-intrinsic-value hippies, so somebody is definitely getting shafted in any given future.
My intuitions expect that in the limit of infinite power, human moral intuitions come apart quite a lot more than you’ve addressed here (for example, I think the utopia described here throws away >90% of the value in the universe for no reason, maybe more depending on how much of the universe gets converted to -oniums).
I also think that any attempt to consider some kind of moral value compromise should probably think about what kinds of process we would actually expect to come up with a nice compromise like this, for example, it seems very unlikely to me that a moral value war would lead to a good ending like this.
(1)
You’re correct that the essay is backchainy. Stratified utopia is my current best bet for “most desirable future given our moral uncertainty” which motivates me to evaluate its likelihood. I don’t think it’s very likely, maybe 5-10%, and I could easily shift with further thought
Starting with the most desirable future and then evaluating its likelihood does risk privileging the hypothesis. This is a fair critique: better epistemics would start with the most likely future and then evaluate its desirability.
(2)
Regarding your example: “if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise.”
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
More common are cases involving contested sacred sites, which also violate the Nothing-To-Mourn Principle. For example, some people would mourn if the Third Temple were never rebuilt on the Temple Mount, while others would mourn if the Al-Aqsa Mosque were destroyed to make way for it.
I agree that this is an empirical claim, in fact, this is pretty much the major interesting question here! Please say more about why you think this is the case. Empirically, Brian Tomasik does exist, as do nature-has-intrinsic-value hippies, so somebody is definitely getting shafted in any given future.
My intuitions expect that in the limit of infinite power, human moral intuitions come apart quite a lot more than you’ve addressed here (for example, I think the utopia described here throws away >90% of the value in the universe for no reason, maybe more depending on how much of the universe gets converted to -oniums).
I also think that any attempt to consider some kind of moral value compromise should probably think about what kinds of process we would actually expect to come up with a nice compromise like this, for example, it seems very unlikely to me that a moral value war would lead to a good ending like this.