I think you may be right that this is what people think of. It seems pretty incompatible with any open source-ish vision of AGI. But what I’m most surprised at, is that people call supervision by humans dystopian/authoritarian, but the same supervision by an ASI (apparently able to see all your data, stop anyone from doing anything, subtly manipulate anyone, etc etc) a utopia. What am I missing here?
Personally, by the way, I imagine a regulation regime to look like regulating a few choke points in the hardware supply chain, plus potentially limits to the hardware or data a person can possess. This doesn’t require an authoritarian regime at all, it’s just regular regulation as we have in many domains already.
In any case, the point was, is something like this going to lead to <=1% xrisk? I think it doesn’t, and definitely not mixed with a democratic/open source AGI vision.
If this is your definition of a dystopia we already live in a dystopia. You can’t make nuclear bombs without being picked up by the FBI/CIA and you’ll probably be arrested in the process. Making something illegal doesn’t define an authoritarian regime. Governments already try to stop international players from building nukes. It just lacks teeth because you can ultimately live with sanctions.
The other problem is it’s way too easy to avoid surveillance or defect in a human regime. For example, you can have a decentralized training network, claim you are training good AIIt’s also unusually easy to regulate AI training. Right now GPUs are easy to control because only Nvidia can make them. This won’t always be true. It’s also much easier to hide GPU production than making nukes because we need GPUs and CPUs for a ton of other useful things.
Theoretically an ASI could probably extrapolate attempts to use compute correctly from your internet signals. Further, if you have the benefits you want from an ASI, you have much less reason to build a 2nd one that’s possibly unaligned. “The digital god says you can’t build it” probably sounds a lot more compelling than “Joe Biden says you can’t build it”.
I think you may be right that this is what people think of. It seems pretty incompatible with any open source-ish vision of AGI. But what I’m most surprised at, is that people call supervision by humans dystopian/authoritarian, but the same supervision by an ASI (apparently able to see all your data, stop anyone from doing anything, subtly manipulate anyone, etc etc) a utopia. What am I missing here?
Personally, by the way, I imagine a regulation regime to look like regulating a few choke points in the hardware supply chain, plus potentially limits to the hardware or data a person can possess. This doesn’t require an authoritarian regime at all, it’s just regular regulation as we have in many domains already.
In any case, the point was, is something like this going to lead to <=1% xrisk? I think it doesn’t, and definitely not mixed with a democratic/open source AGI vision.
If this is your definition of a dystopia we already live in a dystopia. You can’t make nuclear bombs without being picked up by the FBI/CIA and you’ll probably be arrested in the process. Making something illegal doesn’t define an authoritarian regime. Governments already try to stop international players from building nukes. It just lacks teeth because you can ultimately live with sanctions.
The other problem is it’s way too easy to avoid surveillance or defect in a human regime. For example, you can have a decentralized training network, claim you are training good AIIt’s also unusually easy to regulate AI training. Right now GPUs are easy to control because only Nvidia can make them. This won’t always be true. It’s also much easier to hide GPU production than making nukes because we need GPUs and CPUs for a ton of other useful things.
Theoretically an ASI could probably extrapolate attempts to use compute correctly from your internet signals. Further, if you have the benefits you want from an ASI, you have much less reason to build a 2nd one that’s possibly unaligned. “The digital god says you can’t build it” probably sounds a lot more compelling than “Joe Biden says you can’t build it”.