It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
That’s reasonable. We put Plan S in there as an available option because we do think it’s a serious proposal and might even be better than Plan A.
I’ve been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I’ve stopped now.
AI 2040 has given me things to think about.