Contra both the ‘doomers’ and the ‘optimists’ on (not) pausing. Rephrased: RSPs (done right) seem right.
Contra ‘doomers’. Oversimplified, ‘doomers’ (e.g. PauseAI, FLI’s letter, Eliezer) ask(ed) for pausing now / even earlier - (e.g. the Pause Letter). I expect this would be / have been very much suboptimal, even purely in terms of solving technical alignment. For example, Some thoughts on automating alignment research suggests timing the pause so that we can use automated AI safety research could result in ‘[...] each month of lead that the leader started out with would correspond to 15,000 human researchers working for 15 months.’ We clearly don’t have such automated AI safety R&D capabilities now, suggesting that pausing later, when AIs are closer to having the required automated AI safety R&D capabilities would be better. At the same time, current models seem very unlikely to be x-risky (e.g. they’re still very bad at passing dangerous capabilities evals), which is another reason to think pausing now would be premature.
Contra ‘optimists’. I’m more unsure here, but the vibe I’m getting from e.g. AI Pause Will Likely Backfire (Guest Post) is roughly something like ‘no pause ever’; largely based on arguments of current systems seeming easy to align / control. While I agree with the point that current systems do seem easy to align / control and I could even see this holding all the way up to ~human-level automated AI safety R&D, I can easily see scenarios where around that time things get scary quickly without any pause. For example, similar arguments to those about the scalability of automated AI safety R&D suggest automated AI capabilities R&D could also be scaled up significantly. For example, figures like those in Before smart AI, there will be many mediocre or specialized AIs suggest very large populations of ~human-level automated AI capabilities researchers could be deployed (e.g. 100x larger than the current [human] population of AI researchers). Given that even with the current relatively small population, algorithmic progress seems to double LM capabilities ~every 8 months, it seems like algorithmic progress could be much faster with 100x larger populations, potentially leading to new setups (e.g. new AI paradigms, new architectures, new optimizers, synthetic data, etc.) which could quite easily break the properties that make current systems seem relatively easy / safe to align. In this scenario, pausing to get this right (especially since automated AI safety R&D would also be feasible) seems like it could be crucial.
At least Eliezer has been extremely clear that he is in favor of a stop not a pause (indeed, that was like the headline of his article “Pausing AI Developments Isn’t Enough. We Need to Shut it All Down”), so I am confused why you list him with anything related to “pause”.
My guess is me and Eliezer are both in favor of a pause, but mostly because a pause seems like it would slow down AGI progress, not because the next 6 months in-particular will be the most risky period.
At the same time, current models seem very unlikely to be x-risky (e.g. they’re still very bad at passing dangerous capabilities evals), which is another reason to think pausing now would be premature.
The relevant criterion is not whether the current models are likely to be x-risky (it’s obviously far too late if they are!), but whether the next generation of models have more than an insignificant chance of being x-risky together with all the future frameworks they’re likely to be embedded into.
Given that the next generations are planned to involve at least one order of magnitude more computing power in training (and are already in progress!) and that returns on scaling don’t seem to be slowing, I think the total chance of x-risk from those is not insignificant.
Contra both the ‘doomers’ and the ‘optimists’ on (not) pausing. Rephrased: RSPs (done right) seem right.
Contra ‘doomers’. Oversimplified, ‘doomers’ (e.g. PauseAI, FLI’s letter, Eliezer) ask(ed) for pausing now / even earlier - (e.g. the Pause Letter). I expect this would be / have been very much suboptimal, even purely in terms of solving technical alignment. For example, Some thoughts on automating alignment research suggests timing the pause so that we can use automated AI safety research could result in ‘[...] each month of lead that the leader started out with would correspond to 15,000 human researchers working for 15 months.’ We clearly don’t have such automated AI safety R&D capabilities now, suggesting that pausing later, when AIs are closer to having the required automated AI safety R&D capabilities would be better. At the same time, current models seem very unlikely to be x-risky (e.g. they’re still very bad at passing dangerous capabilities evals), which is another reason to think pausing now would be premature.
Contra ‘optimists’. I’m more unsure here, but the vibe I’m getting from e.g. AI Pause Will Likely Backfire (Guest Post) is roughly something like ‘no pause ever’; largely based on arguments of current systems seeming easy to align / control. While I agree with the point that current systems do seem easy to align / control and I could even see this holding all the way up to ~human-level automated AI safety R&D, I can easily see scenarios where around that time things get scary quickly without any pause. For example, similar arguments to those about the scalability of automated AI safety R&D suggest automated AI capabilities R&D could also be scaled up significantly. For example, figures like those in Before smart AI, there will be many mediocre or specialized AIs suggest very large populations of ~human-level automated AI capabilities researchers could be deployed (e.g. 100x larger than the current [human] population of AI researchers). Given that even with the current relatively small population, algorithmic progress seems to double LM capabilities ~every 8 months, it seems like algorithmic progress could be much faster with 100x larger populations, potentially leading to new setups (e.g. new AI paradigms, new architectures, new optimizers, synthetic data, etc.) which could quite easily break the properties that make current systems seem relatively easy / safe to align. In this scenario, pausing to get this right (especially since automated AI safety R&D would also be feasible) seems like it could be crucial.
At least Eliezer has been extremely clear that he is in favor of a stop not a pause (indeed, that was like the headline of his article “Pausing AI Developments Isn’t Enough. We Need to Shut it All Down”), so I am confused why you list him with anything related to “pause”.
My guess is me and Eliezer are both in favor of a pause, but mostly because a pause seems like it would slow down AGI progress, not because the next 6 months in-particular will be the most risky period.
The relevant criterion is not whether the current models are likely to be x-risky (it’s obviously far too late if they are!), but whether the next generation of models have more than an insignificant chance of being x-risky together with all the future frameworks they’re likely to be embedded into.
Given that the next generations are planned to involve at least one order of magnitude more computing power in training (and are already in progress!) and that returns on scaling don’t seem to be slowing, I think the total chance of x-risk from those is not insignificant.