Often, I think about such changes as about phase transitions on a network.
If we assume that these processes (nucleation of clique of the new phase, changes in energy on edge boundaries,...) are independent of the content of moral change, we can expect the emergence of “fluctuations” of new moral phases. Then the question is which of these fluctuations grow to eventually take over the whole network; from an optimistic perspective, this is where relatively small differences between moral phases caused by some phases being “actually better” break the symmetry and lead to gradual moral progress.
Stated in other words, if you look at the micro-dynamic, when you look at the individual edges and nodes, you see the main terms are social pressure, coercion, etc., but the 3rd order terms representing something like “goodness of the moral system in the abstract ” act as a symmetry-breaking term and have large macroscopic consequences.
Turning to longtermism, network-wise, it seems advantageous for the initial bubble of the new phase to spread to central nodes in the network—which seems broadly in line with what EA is doing. Plausibly, in this phase, reasoning plays larger role, and coercion smaller—which is what you see. On the other hand, if longtermism becomes sufficiently large / dominant, I would expect it will become more coercive.
I think this is a good way to think about the issues. My main concerns, put into these terms, are
The network could fall into some super-stable moral phase that’s wrong or far from best. The stability could be enabled by upcoming tech like AI-enabled value lock-in, persuasion, surveillance.
People will get other powers, like being able to create an astronomical number of minds, while the network is still far from the phase that it will eventually settle down to, and use those powers to do things that will turn out to be atrocities when viewed from the right moral philosophy or according to people’s real values.
The random effects overwhelm the directional ones and the network keeps transitioning through various phases far from the best one. (I think this is a less likely outcome though, because it seems like sooner or later it will hit upon one of the super-stable phases mentioned in 1.)
Have you written more about “moral phase transitions” somewhere, or have specific thoughts about these concerns?
Often, I think about such changes as about phase transitions on a network.
If we assume that these processes (nucleation of clique of the new phase, changes in energy on edge boundaries,...) are independent of the content of moral change, we can expect the emergence of “fluctuations” of new moral phases. Then the question is which of these fluctuations grow to eventually take over the whole network; from an optimistic perspective, this is where relatively small differences between moral phases caused by some phases being “actually better” break the symmetry and lead to gradual moral progress.
Stated in other words, if you look at the micro-dynamic, when you look at the individual edges and nodes, you see the main terms are social pressure, coercion, etc., but the 3rd order terms representing something like “goodness of the moral system in the abstract ” act as a symmetry-breaking term and have large macroscopic consequences.
Turning to longtermism, network-wise, it seems advantageous for the initial bubble of the new phase to spread to central nodes in the network—which seems broadly in line with what EA is doing. Plausibly, in this phase, reasoning plays larger role, and coercion smaller—which is what you see. On the other hand, if longtermism becomes sufficiently large / dominant, I would expect it will become more coercive.
I think this is a good way to think about the issues. My main concerns, put into these terms, are
The network could fall into some super-stable moral phase that’s wrong or far from best. The stability could be enabled by upcoming tech like AI-enabled value lock-in, persuasion, surveillance.
People will get other powers, like being able to create an astronomical number of minds, while the network is still far from the phase that it will eventually settle down to, and use those powers to do things that will turn out to be atrocities when viewed from the right moral philosophy or according to people’s real values.
The random effects overwhelm the directional ones and the network keeps transitioning through various phases far from the best one. (I think this is a less likely outcome though, because it seems like sooner or later it will hit upon one of the super-stable phases mentioned in 1.)
Have you written more about “moral phase transitions” somewhere, or have specific thoughts about these concerns?