I worry that the last paragraph of this post is too optimistic.If “formal proof is insufficient”, that might mean that proceeding formally can’t produce superintelligent AI, in which case indeed we don’t need to worry so much about AI risks—but it might instead mean that proceeding formally produces flaky superintelligent AI. That is, AI that’s just about as smart as we’d have hoped or feared, except that it’s extra-vulnerable to malicious hackers, or it has weird cognitive biases a bit like ours but orders of magnitude more subtle, or it basically operates like a “normal” superintelligence except that every now and then a cosmic ray flips a bit and it decides to destroy the sun.
I think the orthodox MIRI position is not that logical proofs are necessary, or even the most efficient way, to make a super-intelligence. It’s that humans need formal proofs to be sure that the AI will be well-behaved. A random kludgy program might be much smarter than your carefully proven one, but that’s cold comfort if it then proceeds to kill you.
I worry that the last paragraph of this post is too optimistic.If “formal proof is insufficient”, that might mean that proceeding formally can’t produce superintelligent AI, in which case indeed we don’t need to worry so much about AI risks—but it might instead mean that proceeding formally produces flaky superintelligent AI. That is, AI that’s just about as smart as we’d have hoped or feared, except that it’s extra-vulnerable to malicious hackers, or it has weird cognitive biases a bit like ours but orders of magnitude more subtle, or it basically operates like a “normal” superintelligence except that every now and then a cosmic ray flips a bit and it decides to destroy the sun.
That would not be reassuring.
Yeah.
I think the orthodox MIRI position is not that logical proofs are necessary, or even the most efficient way, to make a super-intelligence. It’s that humans need formal proofs to be sure that the AI will be well-behaved. A random kludgy program might be much smarter than your carefully proven one, but that’s cold comfort if it then proceeds to kill you.