I should acknowledge that at a low level of intelligence, nihilism (there are no values) may be indistinguishable from fragility-of-value (no value is valuable when considered by itself, but only in combination with other values). In other words, the fact that we can’t exhibit a clearly defined value and explain why (with high probability) this value would be part of one’s CEV can be explained by “value is fragile” as well as by nihilism. So I do not intend to demand that specific proof, and it’s just an example of an argument that would work.
I should acknowledge that at a low level of intelligence, nihilism (there are no values) may be indistinguishable from fragility-of-value (no value is valuable when considered by itself, but only in combination with other values). In other words, the fact that we can’t exhibit a clearly defined value and explain why (with high probability) this value would be part of one’s CEV can be explained by “value is fragile” as well as by nihilism. So I do not intend to demand that specific proof, and it’s just an example of an argument that would work.