So that hinges on a Very Good Question: can we make and contain a potentially Unfriendly Oracle AI without its breaking out and taking over the universe?
To which my answer is: I do not know enough about AGI to answer this question. There are actually loads of advances in AGI remaining before we can make an agent capable of verbal conversation, so it’s difficult to answer.
One approach I might take would be to consider the AI’s “alphabet” of output signals as a programming language, and prove formally that this language can only express safe programs (ie: programs that do not “break out of the box”).
So that hinges on a Very Good Question: can we make and contain a potentially Unfriendly Oracle AI without its breaking out and taking over the universe?
To which my answer is: I do not know enough about AGI to answer this question. There are actually loads of advances in AGI remaining before we can make an agent capable of verbal conversation, so it’s difficult to answer.
One approach I might take would be to consider the AI’s “alphabet” of output signals as a programming language, and prove formally that this language can only express safe programs (ie: programs that do not “break out of the box”).
But don’t quote me on that.