Ramez Naam argues against a hard takeoff by pointing out that we already see recursive self-improvement by superintelligences, such as corporations.
Corporations are not superintelligences. Not in the narrow sense we use when we talk about AGI.
Hall suggests that rather than recursively self-improving its hardware, software, and infrastructure all on its own, a fledgling AI would be better off specializing in one area where it was most effective and then buying the remaining components on the marketplace, because the quality of products on the marketplace continually improves, and the AI would have a hard time keeping up with the cutting-edge technology used by the rest of the world.
The whole idea of superhuman artificial general intelligence is that the machine would be better at everything that humans can do. Also, if the machine wants to specialize, it doesn’t have to deal with humans: it could just create more copies of itself (okay, it would need to buy the hardware, but that’s it) and let different copies specialize at doing different things.
Ben Goertzel agrees with Hall’s suggestion that a new human-level AI would do well to use its intelligence to accumulate wealth.
Not sure if I understand it correctly, but this seems to me like an argument that “wealth = money, therefore the AI will trade with humans”. If that is the intended meaning, I disagree. Money is just one way to get resources. You can also take them by force, steal them, convince people to donate to you, create them, discover them, et cetera. Just because the AI would use its intelligence to gather resources, it does not follow it will trade with humans.
Max More argues that if there were only a few superfast human-level AIs, they wouldn’t radically change the world, because they would still depend on other people to get things done and would still have human cognitive constraints.
Why would the AGI still have human cognitive constraints? Just because we can’t imagine otherwise? Why would it depend on other people to get things done? If it can get some people to build robots, the rest of the work can be done by those robots.
Okay, maybe some of these arguments need much more thinking than I used now, but I wrote this to explain why the arguments in Wikipedia seem completely unimpressive to me. Most of them seem to be based on refusing the idea that anything, including any AI, could really be significantly smarter than humans. “The AI cannot do research in several fields at once, because humans can’t. The AI will not have hands, and will therefore forever depend on humans. The only way an AI could get resources is to have a job, and then buy whatever it needs in the shop. This all ensures that AI is just another human, so other humans will have no problems to overpower it, if needed.”
Quoting Wikipedia:
Corporations are not superintelligences. Not in the narrow sense we use when we talk about AGI.
The whole idea of superhuman artificial general intelligence is that the machine would be better at everything that humans can do. Also, if the machine wants to specialize, it doesn’t have to deal with humans: it could just create more copies of itself (okay, it would need to buy the hardware, but that’s it) and let different copies specialize at doing different things.
Not sure if I understand it correctly, but this seems to me like an argument that “wealth = money, therefore the AI will trade with humans”. If that is the intended meaning, I disagree. Money is just one way to get resources. You can also take them by force, steal them, convince people to donate to you, create them, discover them, et cetera. Just because the AI would use its intelligence to gather resources, it does not follow it will trade with humans.
Why would the AGI still have human cognitive constraints? Just because we can’t imagine otherwise? Why would it depend on other people to get things done? If it can get some people to build robots, the rest of the work can be done by those robots.
Okay, maybe some of these arguments need much more thinking than I used now, but I wrote this to explain why the arguments in Wikipedia seem completely unimpressive to me. Most of them seem to be based on refusing the idea that anything, including any AI, could really be significantly smarter than humans. “The AI cannot do research in several fields at once, because humans can’t. The AI will not have hands, and will therefore forever depend on humans. The only way an AI could get resources is to have a job, and then buy whatever it needs in the shop. This all ensures that AI is just another human, so other humans will have no problems to overpower it, if needed.”