1) Maybe different approximations to logical omniscience work well on different problems. It seems natural to guess that approximations work better overall as they get closer to logical omniscience. We already have a tentative decision theory that works assuming logical omniscience. Can we invent a decision theory that works assuming complete logical ignorance in some sense?
2) We could look at the moment when you decide to make the most general precommitment or self-modification for the future, while still having some logical uncertainty. If you know the first digit of pi but not the millionth, you should precommit accordingly. The first such moment seems to correspond to the AI programmer’s state of logical uncertainty, so if you’re unsure about the millionth digit of pi, then it’s okay for your AI to be unsure.
Just thinking out loud, no complete solution yet.
1) Maybe different approximations to logical omniscience work well on different problems. It seems natural to guess that approximations work better overall as they get closer to logical omniscience. We already have a tentative decision theory that works assuming logical omniscience. Can we invent a decision theory that works assuming complete logical ignorance in some sense?
2) We could look at the moment when you decide to make the most general precommitment or self-modification for the future, while still having some logical uncertainty. If you know the first digit of pi but not the millionth, you should precommit accordingly. The first such moment seems to correspond to the AI programmer’s state of logical uncertainty, so if you’re unsure about the millionth digit of pi, then it’s okay for your AI to be unsure.