One way to deal with this is, have an entire set of utility functions (the different “lines”), normalize them so that they approximately agree inside “Mediocristan”, and choose the “cautious” strategy, i.e. the strategy the maximizes the minimum of expected utility over this set. This way you are at least guaranteed not the end up in a place that is worse than “Mediocristan”.
One way to deal with this is, have an entire set of utility functions (the different “lines”), normalize them so that they approximately agree inside “Mediocristan”, and choose the “cautious” strategy, i.e. the strategy the maximizes the minimum of expected utility over this set. This way you are at least guaranteed not the end up in a place that is worse than “Mediocristan”.
Using PCA on utility functions could be an interesting research subject for wannabe AI risk experts.
So, what utility functions would you give a paperclip maximiser?