What would happen in your GPT-N fusion reactor story if you ask it a broader question about whether it is a good idea to share the plans?
Perhaps relatedly:
>Ok, but can’t we have an AI tell us what questions we need to ask? That’s trainable, right? And we can apply the iterative design loop to make AIs suggest better questions?
I don’t get what your response to this is. Of course, there is the verifiability issue (which I buy). But it seems that the verifiability issue alone is sufficient for failure. If you ask, “Can this design be turned into a bomb?” and the AI says, “No, it’s safe for such and such reasons”, then if you can’t evaluate these reasons, it doesn’t help you that you have asked the right question.
My response to the “get the AI to tell us what questions we need to ask” is that it fails for multiple reasons, any one of which is sufficient for failure. One of them is the verifiability issue. Another is the Gell-Mann Amnesia thing (which you could view as just another frame on the verifiability issue, but up a meta level). Another is the “get what we measure” problem.
Another failure mode which this post did not discuss is the Godzilla Problem. In the frame of this post: in order to work in practice the iterative design loop needs to be able to self-correct; if we make a mistake at one iteration it must be fixable at later iterations. “Get the AI to tell us what questions we need to ask” fails that test; just one iteration of acting on malicious advice from an AI can permanently break the design loop.
Nice post!
What would happen in your GPT-N fusion reactor story if you ask it a broader question about whether it is a good idea to share the plans?
Perhaps relatedly:
>Ok, but can’t we have an AI tell us what questions we need to ask? That’s trainable, right? And we can apply the iterative design loop to make AIs suggest better questions?
I don’t get what your response to this is. Of course, there is the verifiability issue (which I buy). But it seems that the verifiability issue alone is sufficient for failure. If you ask, “Can this design be turned into a bomb?” and the AI says, “No, it’s safe for such and such reasons”, then if you can’t evaluate these reasons, it doesn’t help you that you have asked the right question.
My response to the “get the AI to tell us what questions we need to ask” is that it fails for multiple reasons, any one of which is sufficient for failure. One of them is the verifiability issue. Another is the Gell-Mann Amnesia thing (which you could view as just another frame on the verifiability issue, but up a meta level). Another is the “get what we measure” problem.
Another failure mode which this post did not discuss is the Godzilla Problem. In the frame of this post: in order to work in practice the iterative design loop needs to be able to self-correct; if we make a mistake at one iteration it must be fixable at later iterations. “Get the AI to tell us what questions we need to ask” fails that test; just one iteration of acting on malicious advice from an AI can permanently break the design loop.