Before: The AI is going to take over the world by using its superhuman intelligence to invent nanotechnology.
After: The AI is going to take over the world by inventing nanotechnology.
This trick works only because you already know how exactly the AI going to take over the world, so it doesn’t even need to switch its intelligence on. If, however, you change this for
Before: The AI is going to take over the world by using its superhuman intelligence to do something totally unforeseeable by us meatbags.
After: The AI is going to take over the world by doing something totally unforeseeable by us meatbags.
then there seems to be important distinction. There are lots of things that we can’t foresee, but only some of those would lead the AI to take over the world, and finding this subset would require something of the AI, and we need a name for that something.
Back to the original point, I am quite fond of a metaphor by Marvin Minsky, where he compares intelligence to the concept of “the unexplored regions of Africa,” which makes perfect sense, even though they disappear as soon as we discover them.
This trick works only because you already know how exactly the AI going to take over the world, so it doesn’t even need to switch its intelligence on. If, however, you change this for
Before: The AI is going to take over the world by using its superhuman intelligence to do something totally unforeseeable by us meatbags.
After: The AI is going to take over the world by doing something totally unforeseeable by us meatbags.
then there seems to be important distinction. There are lots of things that we can’t foresee, but only some of those would lead the AI to take over the world, and finding this subset would require something of the AI, and we need a name for that something.
Back to the original point, I am quite fond of a metaphor by Marvin Minsky, where he compares intelligence to the concept of “the unexplored regions of Africa,” which makes perfect sense, even though they disappear as soon as we discover them.