GavinBrown, you haven’t got the hang of the “optimization power in vs. optimized product out” graph. Raw processing power isn’t the crucial element. Sure, such a beast would/will require huge processing power by today’s standards, but the crux is the ability to access to its own code and turn up its optimization ratio on the fly. Once that process has a foothold, it won’t need any more humans to come and put more RAM in a server for it. It won’t need humans at all, which is the concern.
If we knew how to code an AGI, we could do it today and run it on slower machines. Sure, it might take 10-100 times as long to think as machines that we will have in a few years...
This is where a Kurzweil-style graph is useful: that factor of 10-100 would be eaten up in very little time by a multi-level self-optimizing intelligence of the type Eliezer is proposing. Just how quickly, (the rate that the new graph will outstrip the original) I imagine Eliezer has thought very hard about, though I wonder whether he has a firm estimate or just thinks ‘way, way too quickly.’
GavinBrown, you haven’t got the hang of the “optimization power in vs. optimized product out” graph. Raw processing power isn’t the crucial element. Sure, such a beast would/will require huge processing power by today’s standards, but the crux is the ability to access to its own code and turn up its optimization ratio on the fly. Once that process has a foothold, it won’t need any more humans to come and put more RAM in a server for it. It won’t need humans at all, which is the concern.
If we knew how to code an AGI, we could do it today and run it on slower machines. Sure, it might take 10-100 times as long to think as machines that we will have in a few years...
This is where a Kurzweil-style graph is useful: that factor of 10-100 would be eaten up in very little time by a multi-level self-optimizing intelligence of the type Eliezer is proposing. Just how quickly, (the rate that the new graph will outstrip the original) I imagine Eliezer has thought very hard about, though I wonder whether he has a firm estimate or just thinks ‘way, way too quickly.’