Your optimism seems to depend heavily on the difference between “enumerating all proofs up to a certain length and looking for the best provable utility guarantees” and “enumerating proofs until you find a complete set of moral arguments, and behaving randomly if you can’t.”
Yes, and this answers your preceding question:
In response you pointed out that incompleteness isn’t really the problem—there are other failure modes anyway.
The strategy of “enumerating proofs until you find a complete set of moral arguments” doesn’t suffer from the incompleteness issue (whatever it is, if it’s indeed there, which I doubt can have the simple form you referred to).
Why do you believe that a complete set of moral arguments is provable in reasonable situations?
I don’t believe it is provable in any reasonable time, but perhaps given enough time it can often be proven. Building a set of mathematical tools for reasoning about this might prove a fruitful exercise, but I have shelved this line of inquiry for the last few months, and wasn’t working on it.
Yes, and this answers your preceding question:
The strategy of “enumerating proofs until you find a complete set of moral arguments” doesn’t suffer from the incompleteness issue (whatever it is, if it’s indeed there, which I doubt can have the simple form you referred to).
I don’t believe it is provable in any reasonable time, but perhaps given enough time it can often be proven. Building a set of mathematical tools for reasoning about this might prove a fruitful exercise, but I have shelved this line of inquiry for the last few months, and wasn’t working on it.