Arguments that might actually address the cruxes of someone in this reference class might include: [...]
The distribution of outcomes from government interventions are so likely to give you less time, or otherwise make it more difficult to solve the technical alignment problem, that there are fewer surviving worlds where the government intervenes as a result of you asking them to, compared to the counterfactual.
The thing I care more about is quality-adjusted effort, rather than time to solve alignment. For example, I’d generally prefer 30 years to solve alignment with 10 million researchers to 3000 years with 10 researchers, all else being equal. Quality of alignment research comes from a few factors:
How good current AIs are, with the idea being that we’re able to make more progress when testing alignment ideas on AIs that are closer to dangerous-level AGI.
The number of talented people working on the problem, with more generally being better
I expect early delays to lead to negligible additional alignment progress during the delay, relative to future efforts. For example, halting semiconductor production in 2003 for a year to delay AI would have given us almost no additional meaningful alignment progress. I think the same is likely true for 2013 and even 2018. The main impact would just be to delay everything by a year.
In the future I expect to become more optimistic about the merits of delaying AI, but right now I’m not so sure. I think some types of delays might be productive, such as delaying deployment by requiring safety evaluations. But I’m concerned about other types of delays that don’t really give us any meaningful additional quality-adjusted effort.
In particular, the open letter asking for an AI pause appeared to advocate what I consider the worst type of delay: a delay on starting the training of giant models. This type of delay seems least valuable to me for two main reasons.
The first reason is that it wouldn’t significantly slow down algorithmic progress, meaning that after the pause ended, people could likely just go back to training giant models almost like nothing happened. In fact, if people anticipate the pause ending, then they’re likely to invest heavily and then start their training runs on the date the pause ends, which could lead to a significant compute overhang, and thus sudden progress. The second reason is that, compared to a delay of AI deployment, delaying the start of a training run reduces the quality-adjusted effort that AI safety researchers have, as a result of preventing them from testing alignment ideas on more capable models.
If you think that there are non-negligible costs to delaying AI from government action for any reason, then I think it makes sense to be careful about how and when you delay AI, since early and poorly targeted delays may provide negligible benefits. However, I agree that this consideration becomes increasingly less important over time.
The thing I care more about is quality-adjusted effort, rather than time to solve alignment. For example, I’d generally prefer 30 years to solve alignment with 10 million researchers to 3000 years with 10 researchers, all else being equal. Quality of alignment research comes from a few factors:
How good current AIs are, with the idea being that we’re able to make more progress when testing alignment ideas on AIs that are closer to dangerous-level AGI.
The number of talented people working on the problem, with more generally being better
I expect early delays to lead to negligible additional alignment progress during the delay, relative to future efforts. For example, halting semiconductor production in 2003 for a year to delay AI would have given us almost no additional meaningful alignment progress. I think the same is likely true for 2013 and even 2018. The main impact would just be to delay everything by a year.
In the future I expect to become more optimistic about the merits of delaying AI, but right now I’m not so sure. I think some types of delays might be productive, such as delaying deployment by requiring safety evaluations. But I’m concerned about other types of delays that don’t really give us any meaningful additional quality-adjusted effort.
In particular, the open letter asking for an AI pause appeared to advocate what I consider the worst type of delay: a delay on starting the training of giant models. This type of delay seems least valuable to me for two main reasons.
The first reason is that it wouldn’t significantly slow down algorithmic progress, meaning that after the pause ended, people could likely just go back to training giant models almost like nothing happened. In fact, if people anticipate the pause ending, then they’re likely to invest heavily and then start their training runs on the date the pause ends, which could lead to a significant compute overhang, and thus sudden progress. The second reason is that, compared to a delay of AI deployment, delaying the start of a training run reduces the quality-adjusted effort that AI safety researchers have, as a result of preventing them from testing alignment ideas on more capable models.
If you think that there are non-negligible costs to delaying AI from government action for any reason, then I think it makes sense to be careful about how and when you delay AI, since early and poorly targeted delays may provide negligible benefits. However, I agree that this consideration becomes increasingly less important over time.