Which I think seems like a good idea, and is one way to define an objective function which is otherwise very difficult—but as you note, it’s very hard to see how to do it in a practical way.
I hope that at least some researchers work on solving the practical problems. I think it’s a long shot but that’s a lot better than no shot at all.
This seems very similar to coherent extrapolated volition: https://intelligence.org/files/CEV.pdf
Which I think seems like a good idea, and is one way to define an objective function which is otherwise very difficult—but as you note, it’s very hard to see how to do it in a practical way.
I hope that at least some researchers work on solving the practical problems. I think it’s a long shot but that’s a lot better than no shot at all.