The main counterforce IMO is that more and more severe warning shots are quite plausible in worlds where we are doomed, because models are getting destructive faster than they are good at planning (modulo full-blown neuralese/strong continual learning, not mere looping of models.)
(The main caveat is that centaurs/human + AI teams are likely to outperform AIs alone, and thus pure AI warning shots will probably diminish relatively speaking, and the secondary caveat is that some of the good properties are almost certainly lost in future events because of regression to the mean effects, but it’s very plausible there’s a warning shot that’s good enough to make the USG consider an outright pause.)
I suspect this is largely due to models becoming relatively good at SWE/Cyberhacking way faster than with other skills, with the exception of easily-parallelizable mathematics.
The main counterforce IMO is that more and more severe warning shots are quite plausible in worlds where we are doomed, because models are getting destructive faster than they are good at planning (modulo full-blown neuralese/strong continual learning, not mere looping of models.)
(The main caveat is that centaurs/human + AI teams are likely to outperform AIs alone, and thus pure AI warning shots will probably diminish relatively speaking, and the secondary caveat is that some of the good properties are almost certainly lost in future events because of regression to the mean effects, but it’s very plausible there’s a warning shot that’s good enough to make the USG consider an outright pause.)
I suspect this is largely due to models becoming relatively good at SWE/Cyberhacking way faster than with other skills, with the exception of easily-parallelizable mathematics.
I got this point from @davekasten.