The problem that we have with one proposed solution (adding a dummy utility function that highly disvalues a specific non-suffering thing) is that the resulting utility function is not reflectively stable.
So a theory ofvalue formation and especially on achieving vNM coherence (or achieving whatever framework for rational preferences turns out to be the “correct” one) would be useful here. Then during the process of value formation humans can supervise decision points (i.e., in which direction to resolve the preference).
The problem that we have with one proposed solution (adding a dummy utility function that highly disvalues a specific non-suffering thing) is that the resulting utility function is not reflectively stable.
So a theory of value formation and especially on achieving vNM coherence (or achieving whatever framework for rational preferences turns out to be the “correct” one) would be useful here. Then during the process of value formation humans can supervise decision points (i.e., in which direction to resolve the preference).