I wasn’t able to finish that post in the few minutes I’ve got so far today, so here’s the super short version. I remain highly uncertain whether my comments will include any mention of AGI.
I think whether AGI-pilling governments is a good idea is quite complex. Pushing the government to become aware of AGI x-risks will probably decelerate progress, but it could even accelerate it if the conclusion is “build it first, don’t worry we’ll be super careful when we get close”.
Even if it does help with alignment, it’s not necessarily net good. If governments take control early enough to prevent proliferation of AGI, that helps a lot with the risks of misalignment and catastrophic misuse. The US could even cooperate with China to prevent proliferation to other countries and to nongovermental groups, just as the US cooperated with Russia on nuclear nonproliferation.
But government control also raises the risks of power concentration. Intent-aligned AGI in untrustworthy hands could create a permanent dictatorship and unbreakable police state. The current governments of both the US and China don’t seem like the best types to control the future.
This also needs to be balanced agains the possibility of misuse of intent-aligned AGI if it does proliferate broadly; see If we solve alignment, do we die anyway?
If I had a firm estimate of how hard technical alignment is, I’d have a better answer. But I don’t, and I think the best objective conclusion, taking in all of the arguments made to date and the very wide variance in opinion even among those who’ve thought deeply about it, is that nobody has a very good estimate. (Edit: I mean estimates between very very hard and modestly tricky. I don’t know of anyone who’s addressed the hard parts and concluded that it happens by default.)
Neither do we have a good estimate of how likely individuals in power would be to use AGI well or poorly, in various circumstances (unchallenged hegemony vs. close race dynamics).
I wasn’t able to finish that post in the few minutes I’ve got so far today, so here’s the super short version. I remain highly uncertain whether my comments will include any mention of AGI.
(Edit: I finally finished it: Whether governments will control AGI is important and neglected)
I think whether AGI-pilling governments is a good idea is quite complex. Pushing the government to become aware of AGI x-risks will probably decelerate progress, but it could even accelerate it if the conclusion is “build it first, don’t worry we’ll be super careful when we get close”.
Even if it does help with alignment, it’s not necessarily net good. If governments take control early enough to prevent proliferation of AGI, that helps a lot with the risks of misalignment and catastrophic misuse. The US could even cooperate with China to prevent proliferation to other countries and to nongovermental groups, just as the US cooperated with Russia on nuclear nonproliferation.
But government control also raises the risks of power concentration. Intent-aligned AGI in untrustworthy hands could create a permanent dictatorship and unbreakable police state. The current governments of both the US and China don’t seem like the best types to control the future.
So it’s a matter of balancing Fear of centralized power vs. fear of misaligned AGI.
This also needs to be balanced agains the possibility of misuse of intent-aligned AGI if it does proliferate broadly; see If we solve alignment, do we die anyway?
If I had a firm estimate of how hard technical alignment is, I’d have a better answer. But I don’t, and I think the best objective conclusion, taking in all of the arguments made to date and the very wide variance in opinion even among those who’ve thought deeply about it, is that nobody has a very good estimate. (Edit: I mean estimates between very very hard and modestly tricky. I don’t know of anyone who’s addressed the hard parts and concluded that it happens by default.)
Neither do we have a good estimate of how likely individuals in power would be to use AGI well or poorly, in various circumstances (unchallenged hegemony vs. close race dynamics).