What about manipulators? I havent, as far as I know, seen much analysis of manipulation capabilities (and counter-manipulation) on Less Wrong. Mostly there is the AI-box issue (a really freaking big deal, I agree) and then it seems to be considered here that the AI will quickly invent super-nanotech, will not be able to be impeded in its progress, and will become godlike very quickly. I’ve seen some arguments for this, but never a really good analysis, and it’s the remaining reason I am a bit skeptical of the power of FOOM.
The way I think about it, you can set lower bounds on the abilities of an AI by thinking of it as an economic agent. Now, at some point, that abstraction becomes pretty meaningless, but in the early days, a powerful, bootstrapping optimization agent could still incorporate, hire or persuade people to do things for it, make rapid innovations in various fields, have machines made of various types, and generally wind up running the place fairly quickly, even if the problem of bootstrapping versatile nanomachines from current technology turns out to be time-consuming for a superintelligence. I would imagine that nanotech would be where it’d go in the longer run, but that might take time—I don’t know, I don’t know enough about the subject. But even without strong Drexlerian nanotechnology, it’s still possible to get an awful lot done.
What about manipulators? I havent, as far as I know, seen much analysis of manipulation capabilities (and counter-manipulation) on Less Wrong. Mostly there is the AI-box issue (a really freaking big deal, I agree) and then it seems to be considered here that the AI will quickly invent super-nanotech, will not be able to be impeded in its progress, and will become godlike very quickly. I’ve seen some arguments for this, but never a really good analysis, and it’s the remaining reason I am a bit skeptical of the power of FOOM.
The way I think about it, you can set lower bounds on the abilities of an AI by thinking of it as an economic agent. Now, at some point, that abstraction becomes pretty meaningless, but in the early days, a powerful, bootstrapping optimization agent could still incorporate, hire or persuade people to do things for it, make rapid innovations in various fields, have machines made of various types, and generally wind up running the place fairly quickly, even if the problem of bootstrapping versatile nanomachines from current technology turns out to be time-consuming for a superintelligence. I would imagine that nanotech would be where it’d go in the longer run, but that might take time—I don’t know, I don’t know enough about the subject. But even without strong Drexlerian nanotechnology, it’s still possible to get an awful lot done.
That much I do totally agree.