P.S. An additional consideration is the following. Wei Dai’s metaethical alternatives “concentrate on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise (italics mine—S.K.). So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain” However, I am not sure that one can tell apart Wei Dai’s alternative 2 and coexistence of agents with different CEVs merged through acausal bargains.
UPD: What would @Wei Dai say about this approach to ethics and metaethics?
Additionally, there are Wei Dai’s meta-ethical alternatives, and Schelling values would lean into alternative #2
Yes, thanks for the pointer! Quoting Wei’s post for context:
Facts about what everyone should value exist, and most intelligent beings have a part of their mind that can discover moral facts and find them motivating, but those parts don’t have full control over their actions. These beings eventually build or become rational agents with values that represent compromises between different parts of their minds, so most intelligent beings end up having shared moral values along with idiosyncratic values.
One nuance I’d add there is that I think not just randomly true that “most intelligent beings end up having shared moral values along with idiosyncratic values”… I’m pretty that state of affairs is cosmically Schelling good. That is, I think it’s scale-invariantly adaptive for civilizations and beings to have their own notions of “good” that are not fully dictated by the higher scales of organization. This might seem contradictory, but it’s more of a balance than a contradiction. Specifically, it’s a balance between higher and lower scales of organization, similar to balancing exploration and exploitation of strategies.
Thanks, it was the point that I was trying to make in regards to Yudkowsky’s thought experiment with Space Cannibals. Additionally, there are Wei Dai’s meta-ethical alternatives, and Schelling values would lean into alternative #2 or ensure that there is only a finite amount of attractors...
P.S. An additional consideration is the following. Wei Dai’s metaethical alternatives “concentrate on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise (italics mine—S.K.). So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain” However, I am not sure that one can tell apart Wei Dai’s alternative 2 and coexistence of agents with different CEVs merged through acausal bargains.
UPD: What would @Wei Dai say about this approach to ethics and metaethics?
Yes, thanks for the pointer! Quoting Wei’s post for context:
One nuance I’d add there is that I think not just randomly true that “most intelligent beings end up having shared moral values along with idiosyncratic values”… I’m pretty that state of affairs is cosmically Schelling good. That is, I think it’s scale-invariantly adaptive for civilizations and beings to have their own notions of “good” that are not fully dictated by the higher scales of organization. This might seem contradictory, but it’s more of a balance than a contradiction. Specifically, it’s a balance between higher and lower scales of organization, similar to balancing exploration and exploitation of strategies.