This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM
will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.
When a failed test destroys the world, applicability of the normally very useful testing methodology should be reconsidered.
This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
That doesn’t seem like a reliable plan.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.