Arms control for AI in general is therefore unlikely to succeed. The military and civilian applications of general-purpose systems are nearly indistinguishable, and AI will likely see wide use across military and civilian society.
However, the opposite may be true of ASI development control: ASI development would likely be distinguishable from most civilian AI development, and, so long as it is not developed, unintegrated in a state’s economy.
It’s not obvious to me either.
At least in the current paradigm, it seems plausible that a state project of (or, deliberately aimed at) developing ASI would yield a lot of intermediate non-ASI products that would then be dispersed into the economy or military. That’s what we’ve been seeing until now.
Are there reasons to expect this not to continue?
One reason might be that an “ASI Manhattan Project” would want to keep their development secrets so as to minimize information leakage. But would they keep literally all useful intermediate products to themselves? Even if they reveal some X, civilians play with this X, and conclude that X is useless for the purpose of developing ASI, this might still be a valuable negative result that closes off some until-then-plausible ASI development paths.
This is one reason, I think the Manhattan Project is a poor model for a state ASI project. Intermediate results of the original Manhattan Project didn’t trickle down into the economy while the project was still ongoing. I’m not claiming that people are unaware of those disanalogies but I expect thinking in terms of an “ASI Manhattan Project” encourages overanchoring on it.
Pages 22-23:
It’s not obvious to me either.
At least in the current paradigm, it seems plausible that a state project of (or, deliberately aimed at) developing ASI would yield a lot of intermediate non-ASI products that would then be dispersed into the economy or military. That’s what we’ve been seeing until now.
Are there reasons to expect this not to continue?
One reason might be that an “ASI Manhattan Project” would want to keep their development secrets so as to minimize information leakage. But would they keep literally all useful intermediate products to themselves? Even if they reveal some X, civilians play with this X, and conclude that X is useless for the purpose of developing ASI, this might still be a valuable negative result that closes off some until-then-plausible ASI development paths.
This is one reason, I think the Manhattan Project is a poor model for a state ASI project. Intermediate results of the original Manhattan Project didn’t trickle down into the economy while the project was still ongoing. I’m not claiming that people are unaware of those disanalogies but I expect thinking in terms of an “ASI Manhattan Project” encourages overanchoring on it.