I simply do not think that this produces something that I would consider a superintelligence. I don’t think you can use this approach to mimic the intelligence of a single human mathematician 12 SD above the mean, in the same way you can’t organise IQ 100 humans to mimic Jon Von Neumann (a placeholder for a recognised genius that was almost certainly within 7 SD of the mean).
Collection and organisation is just not a valuable path to superintelligence, because aggregate intelligence of a committee doesn’t scale gracefully.
I think there’s an interesting question of whether or not you need 12 SD to end the “acute risk period”, e.g. by inventing nanotechnology.
It’s not implausible to me that you can take 100 5-SD-humans, run them for 1000 subjective years to find a more ambitious solution to the alignment problem or a manual for nanotechnology, and thus end the acute risk period. I admittedly don’t have domain insight into the difficulty of nanotech, but I was not under the impression that it was non-computable in this sense.
Aggregation may not scale gracefully, but extra time does (and time tends to be the primary resource cost in increasing bureaucracy size).
I simply do not think that this produces something that I would consider a superintelligence. I don’t think you can use this approach to mimic the intelligence of a single human mathematician 12 SD above the mean, in the same way you can’t organise IQ 100 humans to mimic Jon Von Neumann (a placeholder for a recognised genius that was almost certainly within 7 SD of the mean).
Collection and organisation is just not a valuable path to superintelligence, because aggregate intelligence of a committee doesn’t scale gracefully.
I think there’s an interesting question of whether or not you need 12 SD to end the “acute risk period”, e.g. by inventing nanotechnology.
It’s not implausible to me that you can take 100 5-SD-humans, run them for 1000 subjective years to find a more ambitious solution to the alignment problem or a manual for nanotechnology, and thus end the acute risk period. I admittedly don’t have domain insight into the difficulty of nanotech, but I was not under the impression that it was non-computable in this sense.
Aggregation may not scale gracefully, but extra time does (and time tends to be the primary resource cost in increasing bureaucracy size).