(1) The difficulty of specifying what it means for a superintelligent system to only have a limited or bounded effect.
(2) The difficulty of getting an oracle AI to “just tell the truth” when the questions you ask it don’t have a simple, straight answer that can be told, and where the choices about what to say will have significant effects upon the world such that you would be better off explicitly having a sovereign AI because your oracle is now moonlighting as a sovereign when it chooses how to answer.
The fundamental problems with “tool AI” are
(1) The difficulty of specifying what it means for a superintelligent system to only have a limited or bounded effect.
(2) The difficulty of getting an oracle AI to “just tell the truth” when the questions you ask it don’t have a simple, straight answer that can be told, and where the choices about what to say will have significant effects upon the world such that you would be better off explicitly having a sovereign AI because your oracle is now moonlighting as a sovereign when it chooses how to answer.