As I see it, the central path to 50 trillion supergeniuses thinking at 10 000x speed in the asteroid belt passes through a scenario of comparable difficulty to 50 million geniuses at 10x speed in datacentres.
It’s a relevant scenario because we’re almost certainly going to face it soon, and it involves AI capable enough that it’s highly believable that we would lose control (and/or life). The millions of instances and moderate speed increase over human aren’t arbitrary numbers either, they’re roughly the ratios we get from existing plans.
If we can’t deal with the country-level scenario, it makes no difference whether or not we can deal with trillions of offworld supergeniuses. If AI genius countries do actually cooperate with us, that’s a huge step toward being able to consider whether we—and they—could or should allow the latter scenario.
The country-level scenario could be bypassed by a recursive self-improving singleton that does find some weird tricks, but many of the solutions to the sorts of problems we would face with fast geniuses in datacentres should also help prevent runaway RSI singletons.
As I see it, the central path to 50 trillion supergeniuses thinking at 10 000x speed in the asteroid belt passes through a scenario of comparable difficulty to 50 million geniuses at 10x speed in datacentres.
It’s a relevant scenario because we’re almost certainly going to face it soon, and it involves AI capable enough that it’s highly believable that we would lose control (and/or life). The millions of instances and moderate speed increase over human aren’t arbitrary numbers either, they’re roughly the ratios we get from existing plans.
If we can’t deal with the country-level scenario, it makes no difference whether or not we can deal with trillions of offworld supergeniuses. If AI genius countries do actually cooperate with us, that’s a huge step toward being able to consider whether we—and they—could or should allow the latter scenario.
The country-level scenario could be bypassed by a recursive self-improving singleton that does find some weird tricks, but many of the solutions to the sorts of problems we would face with fast geniuses in datacentres should also help prevent runaway RSI singletons.