Well, it would be helpful if we could also: 2.5) work out a reliable test for whether a given X really is an instance of the CEV concept for the given reference class
Which seems to depend on having some kind of understanding.
Lacking that, we are left with having to trust that whatever the SI we’ve built is doing is actually what we “really want” it to do, even if we don’t seem to want it to do that, which is an awkward place to be.
I think this doesn’t matter, if we can
1) successfully define the CEV concept itself,
2) define a suitable reference class,
3) build a superintelligence, and
4) ensure that the superintelligence continues to pursue the best CEV it can find for the appropriate reference class.
Well, it would be helpful if we could also:
2.5) work out a reliable test for whether a given X really is an instance of the CEV concept for the given reference class
Which seems to depend on having some kind of understanding.
Lacking that, we are left with having to trust that whatever the SI we’ve built is doing is actually what we “really want” it to do, even if we don’t seem to want it to do that, which is an awkward place to be.