One problem that could cause the searching process to be unsafe is if the prior contained a relatively large measure of malign agents. This could happen if you used the universal prior, per Paul’s argument. Such agents could maximize across the propositions you test them on, but do something else once they think they’re deployed.
If the prior is full of malign agents, then you are selecting your new logical optimizer based on its ability to correctly answer arbitrary questions (in a certain format) about malign agents. This doesn’t seem to be that problematic. If the set of programs being logically optimized over is malign, then you have trouble.
One problem that could cause the searching process to be unsafe is if the prior contained a relatively large measure of malign agents. This could happen if you used the universal prior, per Paul’s argument. Such agents could maximize across the propositions you test them on, but do something else once they think they’re deployed.
If the prior is full of malign agents, then you are selecting your new logical optimizer based on its ability to correctly answer arbitrary questions (in a certain format) about malign agents. This doesn’t seem to be that problematic. If the set of programs being logically optimized over is malign, then you have trouble.
Ah, sorry, I misread the terminology. I agree.