Suppose you make a super-intelligent AI and run it on a computer. The computer has NO conventional means of output (no connections to other computers, no screen, etc).
Indeed. What’s the point of building an AI you’re never going to communicate with?
Also, you can’t build it that way. Programs never work the first time, so at a minimum you’re going to have a long period of time where programmers are coding, testing and debugging various parts of the AI. As it nears completion that’s going to involve a great deal of unsupervised interaction with a partially-functional AI, because without interaction you can’t tell if it works.
So what are you going to do? Wait until the AI is feature-complete on day X, and then box it? Do you really think the AI was safe on day X-1, when it just had a couple of little bugs left? How about on day X-14, when you thought the major systems were all working but there was actually a major bug in the expected utility calculator? Or on day X-60, when a programmer got the Bayesian reasoning system working but it was connected to a stubbed-out version of the goal system instead of the real thing?
This myopic focus on boxing ideas misses most of the problems inherent in building a safe AGI.
Why would anyone do that?
Indeed. What’s the point of building an AI you’re never going to communicate with?
Also, you can’t build it that way. Programs never work the first time, so at a minimum you’re going to have a long period of time where programmers are coding, testing and debugging various parts of the AI. As it nears completion that’s going to involve a great deal of unsupervised interaction with a partially-functional AI, because without interaction you can’t tell if it works.
So what are you going to do? Wait until the AI is feature-complete on day X, and then box it? Do you really think the AI was safe on day X-1, when it just had a couple of little bugs left? How about on day X-14, when you thought the major systems were all working but there was actually a major bug in the expected utility calculator? Or on day X-60, when a programmer got the Bayesian reasoning system working but it was connected to a stubbed-out version of the goal system instead of the real thing?
This myopic focus on boxing ideas misses most of the problems inherent in building a safe AGI.