If you read the scenario, you’ll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I’m tempted to say, “will probably discover”) and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I’m not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
I’m not sure that actual existing efforts towards delaying AGI are helping. I think the path from here to AGI is bottlenecked by researchers playing with toy models, and publishing stuff on arXiv and GitHub. … Once the new paradigm is known and developed, the actors able to train ASI from scratch will probably number in the tens of thousands, spread all around the world. We’re not just talking about five giant firms with gazillion-dollar data centers, as LLM-focused people tend to imagine.
Given the dire global situation, it seems logical that there should be loud, sustained and coherent calls for restrictions on teaching and publishing about cutting-edge learning algorithms. Ever since WW II, Washington has restricted what experts can publish or teach about sensitive military applications of radar technology. Violations of these restrictions, such as those governed by the Invention Secrecy Act and military classification laws, carry severe consequences. Experts on nuclear weaponry face even stricter controls on information dissemination.
Although there is not currently enough political will to enact restrictions on the dissemination of advances in machine learning, the global situation is dire enough that our most effective course of action is to put most of our energy into planning and preparing for unlikely timelines in which we get lucky (i.e., playing to our outs). One of the most likely ways we get lucky is if there is a rapid appearance of enough political will to slow down the AI juggernaut. If enough political will manifests, we must be ready with a plan that can actually save us. In my humble opinion, the AI 2040 document’s Plan A falls short of being such a plan because it does nothing to stop or slow research that might find (and I’m tempted to say, “will probably find”) a fundamentally more efficient learning algorithm than what the leading labs are currently using.
In response to my comment, Mr Kokotajlo would probably point out that Washington’s purpose in restricting the dissemination of detailed radar technology information was to prevent it from falling into the hands of foreign powers. Therefore, putting those same restrictions on AI knowledge would cause Beijing and other governments to increase their investment in AI research. In turn, I would respond that I mentioned the restrictions on radar technology only to show that restrictions on technology can be effective and feasible over the long term. The purpose of the AI restrictions would be different. They would aim to slow down AI research and, to the extent possible, channel it toward designs that are easier to align. Consequently, the regulations would take a different shape. Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa. Also, the regulations would seek to restrict advanced AI knowledge to researchers who have obtained a security clearance (i.e., have been investigated to verify that the researcher lives a stable life, is not a political extremist and can’t easily be blackmailed) and have signed documents confirming that they understand that dissemination of the knowledge is a serious crime. I.e., almost the complete opposite of Kokotaljo’s proposal that “the ideas themselves are required to be totally transparent to the public”.
Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
In summary, can we please stop writing as if all that is needed to avert our extinction is to put sufficient restrictions on the world’s giant data centers?
Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
Plan A largely doesn’t agree with this. Since they expect the slowdown treaty to eventually fall apart—at which point we revert to the status quo arms race, which we all agree is bad—they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
In my humble opinion, the AI 2040 document falls short of being such a plan because it does nothing to stop or slow research that might find (and I’m tempted to say, “will probably find”) a fundamentally more efficient learning algorithm than what the leading labs are currently using.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
In the past, companies have trained bigger and better AIs using both compute scaling (bigger training runs) and software progress (advances in AI algorithms—new paradigms, better training recipes, better data, etc.). Now, the Consortium tries to steer things so that the majority of improvement comes from increasing training compute.
Algorithms are information; it’s inherently difficult to stop them from proliferating, and the total research transparency means we aren’t even trying. [FN: There are a few nuances here. Some algorithmic progress is easily communicated (e.g., new architectures, better optimization algorithms), while other types cannot easily diffuse (e.g., huge libraries of RL environments, hardware-software codesign, scale or compute dependent algorithms). Regulations that the US and China agree to should steer companies towards making the latter type of algorithm progress when possible.] Once a new paradigm is discovered, it’ll go straight to the hypothetical covert projects and there’s no way to undo that.
and:
So the goal is to develop AI at a reasonable pace: Not too fast, not too slow. Ban the unsafe kinds, allow the safe kinds. (...) The Consortium countries come up with a simple high-level framework: we’ll agree to let each other see all the AI research. Then, if we don’t like something someone is doing, we’ll talk about it and perhaps agree to ban it. (...)
Transparency offers many benefits. (...) Third, there’s no longer as much incentive to race to discover new AI paradigms and more powerful algorithms, because companies wouldn’t be able to hoard such discoveries and profit greatly from them. This buys the world time.
Plan A involves significant attempts to slow down algorithmic progress
Plan A does not advocate for what I consider the most potent brake on that “progress”, namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
there’s no longer as much incentive to race to discover new AI paradigms and more powerful algorithms, because companies wouldn’t be able to hoard such discoveries and profit greatly from them.
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind’s freely giving away the knowledge described in the “Attention is all you need” paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the history of AI research, I know enough about the history of research and development in general to say with confidence that at least over the last 200 years, the vast majority of technological and scientific progress over all fields was caused by these acts of freely giving away insights. I’d be very surprised to learn that AI is an exception to that generality: the vast amount of potential revenue and profit to be made with AI makes giving-away less likely, but the idealism and (false) sense of an altruistic purpose makes giving-away more likely.
To slow down the rate of advance of a field, we would prevent these acts of giving-away from happening: if we could somehow prevent all insights discovered inside OpenAI from disseminating beyond OpenAI, that would IMHO be an improvement (though shutting OpenAI down and requiring all its employees to find something else to do other than AI research would be strictly better)!
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
If AI researchers continue to be able to communicate without restriction, the research community might discover (and I’m tempted to say, “will probably discover”) and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware.
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain’s plasticity and it wouldn’t matter if it doesn’t also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.
Mr Kokotajlo, one of the authors of this AI 2040 scenario, describes it as follows:
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I’m tempted to say, “will probably discover”) and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I’m not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
Given the dire global situation, it seems logical that there should be loud, sustained and coherent calls for restrictions on teaching and publishing about cutting-edge learning algorithms. Ever since WW II, Washington has restricted what experts can publish or teach about sensitive military applications of radar technology. Violations of these restrictions, such as those governed by the Invention Secrecy Act and military classification laws, carry severe consequences. Experts on nuclear weaponry face even stricter controls on information dissemination.
Although there is not currently enough political will to enact restrictions on the dissemination of advances in machine learning, the global situation is dire enough that our most effective course of action is to put most of our energy into planning and preparing for unlikely timelines in which we get lucky (i.e., playing to our outs). One of the most likely ways we get lucky is if there is a rapid appearance of enough political will to slow down the AI juggernaut. If enough political will manifests, we must be ready with a plan that can actually save us. In my humble opinion, the AI 2040 document’s Plan A falls short of being such a plan because it does nothing to stop or slow research that might find (and I’m tempted to say, “will probably find”) a fundamentally more efficient learning algorithm than what the leading labs are currently using.
In response to my comment, Mr Kokotajlo would probably point out that Washington’s purpose in restricting the dissemination of detailed radar technology information was to prevent it from falling into the hands of foreign powers. Therefore, putting those same restrictions on AI knowledge would cause Beijing and other governments to increase their investment in AI research. In turn, I would respond that I mentioned the restrictions on radar technology only to show that restrictions on technology can be effective and feasible over the long term. The purpose of the AI restrictions would be different. They would aim to slow down AI research and, to the extent possible, channel it toward designs that are easier to align. Consequently, the regulations would take a different shape. Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa. Also, the regulations would seek to restrict advanced AI knowledge to researchers who have obtained a security clearance (i.e., have been investigated to verify that the researcher lives a stable life, is not a political extremist and can’t easily be blackmailed) and have signed documents confirming that they understand that dissemination of the knowledge is a serious crime. I.e., almost the complete opposite of Kokotaljo’s proposal that “the ideas themselves are required to be totally transparent to the public”.
Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
In summary, can we please stop writing as if all that is needed to avert our extinction is to put sufficient restrictions on the world’s giant data centers?
Plan A largely doesn’t agree with this. Since they expect the slowdown treaty to eventually fall apart—at which point we revert to the status quo arms race, which we all agree is bad—they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Scott Alexander had a nice little graphic on this point in his ACX post on Plan A:
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
and:
Plan A does not advocate for what I consider the most potent brake on that “progress”, namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind’s freely giving away the knowledge described in the “Attention is all you need” paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the history of AI research, I know enough about the history of research and development in general to say with confidence that at least over the last 200 years, the vast majority of technological and scientific progress over all fields was caused by these acts of freely giving away insights. I’d be very surprised to learn that AI is an exception to that generality: the vast amount of potential revenue and profit to be made with AI makes giving-away less likely, but the idealism and (false) sense of an altruistic purpose makes giving-away more likely.
To slow down the rate of advance of a field, we would prevent these acts of giving-away from happening: if we could somehow prevent all insights discovered inside OpenAI from disseminating beyond OpenAI, that would IMHO be an improvement (though shutting OpenAI down and requiring all its employees to find something else to do other than AI research would be strictly better)!
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
That’s reasonable. We put Plan S in there as an available option because we do think it’s a serious proposal and might even be better than Plan A.
I’ve been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I’ve stopped now.
AI 2040 has given me things to think about.
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain’s plasticity and it wouldn’t matter if it doesn’t also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.