I think there’s an interesting question of whether or not you need 12 SD to end the “acute risk period”, e.g. by inventing nanotechnology.
It’s not implausible to me that you can take 100 5-SD-humans, run them for 1000 subjective years to find a more ambitious solution to the alignment problem or a manual for nanotechnology, and thus end the acute risk period. I admittedly don’t have domain insight into the difficulty of nanotech, but I was not under the impression that it was non-computable in this sense.
Aggregation may not scale gracefully, but extra time does (and time tends to be the primary resource cost in increasing bureaucracy size).
I think there’s an interesting question of whether or not you need 12 SD to end the “acute risk period”, e.g. by inventing nanotechnology.
It’s not implausible to me that you can take 100 5-SD-humans, run them for 1000 subjective years to find a more ambitious solution to the alignment problem or a manual for nanotechnology, and thus end the acute risk period. I admittedly don’t have domain insight into the difficulty of nanotech, but I was not under the impression that it was non-computable in this sense.
Aggregation may not scale gracefully, but extra time does (and time tends to be the primary resource cost in increasing bureaucracy size).