The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.