I think most of the arguments in this essay fail to bind to reality. This essay seems to have been backchained from a cool idea for the future into arguments for it. The points about how “The cosmos will be divided between different value systems” are all really vague and don’t really provide any insight into what the future actually looks like, or how any of these processes really lead to a future like the one described, yet the descriptions of each individual layer are very specific.
(I can imagine that maybe after some kind of long reflection, we all agree on something like this, but I expect that any actual war scenarios end up with a winner-takes-all lock-in)
I do think the intuitions that a stratified utopia is desirable are somewhat interesting. I think that dividing up the universe into various chunks probably is a way to create a future that most people would be happy with. The “Nothing to Mourn” principle is really nice.
Then again, I think that a simple forward-chaining application of the nothing to mourn principle immediately runs into the real, difficult problems of allocating resources to different values: some people’s utopias are mutually net negative. For example, if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise. It’s not possible. This is the real challenge of allocating value through splitting up the universe, and the fact that you didn’t really address it gives the whole essay a kind of “Communist art students planning their lives on the commune after the revolution” vibe.
It would be cool to do a dive into this concept which focuses more on what kind of a thing a value actually is, and what moral uncertainty actually means (some people especially EAs do this thing where they talk about moral uncertainty as if they’re moral realists, but firstly I think moral realism is incoherent, and secondly they don’t actually endorse moral realism) and also to address the problem of mutually net negative ideal worlds.
You’re correct that the essay is backchainy. Stratified utopia is my current best bet for “most desirable future given our moral uncertainty” which motivates me to evaluate its likelihood. I don’t think it’s very likely, maybe 5-10%, and I could easily shift with further thought
Starting with the most desirable future and then evaluating its likelihood does risk privileging the hypothesis. This is a fair critique: better epistemics would start with the most likely future and then evaluate its desirability.
(2)
Regarding your example: “if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise.”
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
More common are cases involving contested sacred sites, which also violate the Nothing-To-Mourn Principle. For example, some people would mourn if the Third Temple were never rebuilt on the Temple Mount, while others would mourn if the Al-Aqsa Mosque were destroyed to make way for it.
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
I agree that this is an empirical claim, in fact, this is pretty much the major interesting question here! Please say more about why you think this is the case. Empirically, Brian Tomasik does exist, as do nature-has-intrinsic-value hippies, so somebody is definitely getting shafted in any given future.
My intuitions expect that in the limit of infinite power, human moral intuitions come apart quite a lot more than you’ve addressed here (for example, I think the utopia described here throws away >90% of the value in the universe for no reason, maybe more depending on how much of the universe gets converted to -oniums).
I also think that any attempt to consider some kind of moral value compromise should probably think about what kinds of process we would actually expect to come up with a nice compromise like this, for example, it seems very unlikely to me that a moral value war would lead to a good ending like this.
I think most of the arguments in this essay fail to bind to reality. This essay seems to have been backchained from a cool idea for the future into arguments for it. The points about how “The cosmos will be divided between different value systems” are all really vague and don’t really provide any insight into what the future actually looks like, or how any of these processes really lead to a future like the one described, yet the descriptions of each individual layer are very specific.
(I can imagine that maybe after some kind of long reflection, we all agree on something like this, but I expect that any actual war scenarios end up with a winner-takes-all lock-in)
I do think the intuitions that a stratified utopia is desirable are somewhat interesting. I think that dividing up the universe into various chunks probably is a way to create a future that most people would be happy with. The “Nothing to Mourn” principle is really nice.
Then again, I think that a simple forward-chaining application of the nothing to mourn principle immediately runs into the real, difficult problems of allocating resources to different values: some people’s utopias are mutually net negative. For example, if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise. It’s not possible. This is the real challenge of allocating value through splitting up the universe, and the fact that you didn’t really address it gives the whole essay a kind of “Communist art students planning their lives on the commune after the revolution” vibe.
It would be cool to do a dive into this concept which focuses more on what kind of a thing a value actually is, and what moral uncertainty actually means (some people especially EAs do this thing where they talk about moral uncertainty as if they’re moral realists, but firstly I think moral realism is incoherent, and secondly they don’t actually endorse moral realism) and also to address the problem of mutually net negative ideal worlds.
(1)
You’re correct that the essay is backchainy. Stratified utopia is my current best bet for “most desirable future given our moral uncertainty” which motivates me to evaluate its likelihood. I don’t think it’s very likely, maybe 5-10%, and I could easily shift with further thought
Starting with the most desirable future and then evaluating its likelihood does risk privileging the hypothesis. This is a fair critique: better epistemics would start with the most likely future and then evaluate its desirability.
(2)
Regarding your example: “if one person thinks the natural world is a horrid hell-pit of suffering and another thinks that living in a fully AI-managed environment is a kind of torture for everyone involved, they just can’t compromise.”
I should clarify that resource-compatibility is a claim about the mundane and exotic values humans actually hold. It’s a contingent, not a necessary. Yes, some people think the natural world is a hell-pit of suffering (negative utilitarians like Brian Tomasik), but they’re typically scope-sensitive and longtermist, so they’d care far more about the distal resources.
You could construct a value profile like “utility = −1 if suffering exists on Earth, else 0” which is exotic values seeking proximal resources. I don’t have a good answer for handling such cases. But empirically, this value profile seems rare.
More common are cases involving contested sacred sites, which also violate the Nothing-To-Mourn Principle. For example, some people would mourn if the Third Temple were never rebuilt on the Temple Mount, while others would mourn if the Al-Aqsa Mosque were destroyed to make way for it.
I agree that this is an empirical claim, in fact, this is pretty much the major interesting question here! Please say more about why you think this is the case. Empirically, Brian Tomasik does exist, as do nature-has-intrinsic-value hippies, so somebody is definitely getting shafted in any given future.
My intuitions expect that in the limit of infinite power, human moral intuitions come apart quite a lot more than you’ve addressed here (for example, I think the utopia described here throws away >90% of the value in the universe for no reason, maybe more depending on how much of the universe gets converted to -oniums).
I also think that any attempt to consider some kind of moral value compromise should probably think about what kinds of process we would actually expect to come up with a nice compromise like this, for example, it seems very unlikely to me that a moral value war would lead to a good ending like this.