For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!
It’s called AI 2040: Plan A.
It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.
It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.
As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.
Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer.
What’s next for us?
Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that AI 2040: Plan A sparks. Beyond that, we aren’t sure yet. In general our mission is to help make AGI go well, and now we’ve tried out both forecasting and planning. Maybe we’ll get started on another big scenario. On the other hand, these megaprojects take so much time…
Mr Kokotajlo, one of the authors of this AI 2040 scenario, describes it as follows:
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I’m tempted to say, “will probably discover”) and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I’m not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
Given the dire global situation, it seems logical that there should be loud, sustained and coherent calls for restrictions on teaching and publishing about cutting-edge learning algorithms. Ever since WW II, Washington has restricted what experts can publish or teach about sensitive military applications of radar technology. Violations of these restrictions, such as those governed by the Invention Secrecy Act and military classification laws, carry severe consequences. Experts on nuclear weaponry face even stricter controls on information dissemination.
Although there is not currently enough political will to enact restrictions on the dissemination of advances in machine learning, the global situation is dire enough that our most effective course of action is to put most of our energy into planning and preparing for unlikely timelines in which we get lucky (i.e., playing to our outs). One of the most likely ways we get lucky is if there is a rapid appearance of enough political will to slow down the AI juggernaut. If enough political will manifests, we must be ready with a plan that can actually save us. In my humble opinion, the AI 2040 document’s Plan A falls short of being such a plan because it does nothing to stop or slow research that might find (and I’m tempted to say, “will probably find”) a fundamentally more efficient learning algorithm than what the leading labs are currently using.
In response to my comment, Mr Kokotajlo would probably point out that Washington’s purpose in restricting the dissemination of detailed radar technology information was to prevent it from falling into the hands of foreign powers. Therefore, putting those same restrictions on AI knowledge would cause Beijing and other governments to increase their investment in AI research. In turn, I would respond that I mentioned the restrictions on radar technology only to show that restrictions on technology can be effective and feasible over the long term. The purpose of the AI restrictions would be different. They would aim to slow down AI research and, to the extent possible, channel it toward designs that are easier to align. Consequently, the regulations would take a different shape. Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa. Also, the regulations would seek to restrict advanced AI knowledge to researchers who have obtained a security clearance (i.e., have been investigated to verify that the researcher lives a stable life, is not a political extremist and can’t easily be blackmailed) and have signed documents confirming that they understand that dissemination of the knowledge is a serious crime. I.e., almost the complete opposite of Kokotaljo’s proposal that “the ideas themselves are required to be totally transparent to the public”.
Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
In summary, can we please stop writing as if all that is needed to avert our extinction is to put sufficient restrictions on the world’s giant data centers?
Plan A largely doesn’t agree with this. Since they expect the slowdown treaty to eventually fall apart—at which point we revert to the status quo arms race, which we all agree is bad—they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Scott Alexander had a nice little graphic on this point in his ACX post on Plan A:
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
and:
Plan A does not advocate for what I consider the most potent brake on that “progress”, namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind’s freely giving away the knowledge described in the “Attention is all you need” paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the history of AI research, I know enough about the history of research and development in general to say with confidence that at least over the last 200 years, the vast majority of technological and scientific progress over all fields was caused by these acts of freely giving away insights. I’d be very surprised to learn that AI is an exception to that generality: the vast amount of potential revenue and profit to be made with AI makes giving-away less likely, but the idealism and (false) sense of an altruistic purpose makes giving-away more likely.
To slow down the rate of advance of a field, we would prevent these acts of giving-away from happening: if we could somehow prevent all insights discovered inside OpenAI from disseminating beyond OpenAI, that would IMHO be an improvement (though shutting OpenAI down and requiring all its employees to find something else to do other than AI research would be strictly better)!
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
That’s reasonable. We put Plan S in there as an available option because we do think it’s a serious proposal and might even be better than Plan A.
I’ve been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I’ve stopped now.
AI 2040 has given me things to think about.
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain’s plasticity and it wouldn’t matter if it doesn’t also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.
“I wish it need not have happened in my time,” said Frodo. “So do I,” said Gandalf, “and so do all who live to see such times.”
Is there any reason you haven’t included human intelligence enhancement as part of scenario S?
Given the level of political will and international coordination in the story, why can’t they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made—something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn’t have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A.
With the compute supply chain dismantled, the pause deal falling apart looks much less worrying. I expect that even if everyone started to try going full-speed when the deal fails, it would still take 10-20 years to rebuild the compute supply chain and train cutting-edge AIs again. During those 10-20 years, the pause deal can be revived again, and generally it seems good to have this longer time.
So I think the biggest question here is whether we expect the world to generally get better or worse during a long pause. I expect that the world is probably going to get better: I find the long-term historical trends encouraging, and I think that if we try to build AGI a hundred years from now, that will likely go better than if we try now. (Especially if we do some genetic engineering in the meantime.)
If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
I think countries sometimes have honor and follow their commitments even if it’s not locally game theoretically optimal.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
I think it’s pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn’t restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don’t believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
See https://ai-2040.com/supplements/deal-decline for my probabilistic estimates here.
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there’s probably other ways) is if you think that most of the value of “~infinity growth now” would already have been lost by moving to “~infinity growth in 10 years”. But that doesn’t seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than “singularity in 10 years” by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
Yes, that’s fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways. (E.g. I don’t expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge—I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there’s much less hope of that if there’s an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn’t that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don’t currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they’ll have that big of an effect?
It’s not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn’t explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn’t Significantly Enhancing Adult Intelligence With Gene Editing Possible?
What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we’ll have a powerful AI capable of generating better ideas): “If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we’ve effectively boosted someone’s brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.” (https://www.lesswrong.com/s/Wy8smXGZ7PsS9CpgP/p/ewZXQgzaCvzdSvtWE)
What is Michael Levin doing with his bioelectricity? Yudkowsky considers work on enhancing adult intelligence the most important priority after an AI pause — so what are he and MIRI actually doing on that front? What about the field of NeuroAI (Astera Institute, Allen Institute, Amaranth.foundation)?
My point is that fundamental breakthroughs in all these fields are turning the tide. And when we have 10+ years and a powerful AI, I think there’s a pretty good chance that some of this will work BEFORE AI takes off.
kman here. I should mention that I now think that post was very overly optimistic, and I don’t think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn’t seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as “scenario S”.
(I still think adult enhancement is worth pursuing as well. I’m currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it’ll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
(See also kman’s comment.) Besides the technical difficulties with trying the experiments, it’s quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on “crystallized intelligence”; to go bigger, I’m guessing you want more child-like learning capabilities; though that’s just speculation.) Even for IQ genes that are expressed in adult brains, it’s plausible their effects on IQ are mainly or linearly coming from childhood effects.
My uninformed impression of E11 is that it’s very cool, but also is not the bottleneck to WBE or similar. The connectome isn’t the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
I don’t believe the claim about simulating something close enough to neurons; see the previous point. Re/ BCIs, see also: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=ff9c8PHtuFKfwE6Yj
I highly doubt this is relevant.
In general there is the issue that it’s hard to test what you’re interested in; and you’re not constrained to the natural manifold (as with reprogenetics), so there’s little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
I agree that BCIs probably wouldn’t do that much. (Though:) There are a few threads that one can imagine, such as:
networking people together and having them spend a bunch of time learning how to use that;
prosthetically increased long-range connectivity;
some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
networking with some sort of neural organoids to get “more cortex”.
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I’m not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like “do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes”. It’s hard to tell and I think it’s unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
If that’s a crux, happy to give more info.
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
I agree with the overall point! We tried to think through a bunch of technological progress but no doubt were missing some important stuff.
See, for example:
- Military power
- Robots (mentioned throughout the secnario)
- Lie detectors
I haven’t looked into BCIs much at all it’s plausible they should play a bigger role earlier.
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the “you’re a everyday citizen” scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)
Total research transparency feels a bit too galaxy-brained for me. It makes non-robust assumptions that newly discovered techniques won’t be usable to enhance already existing open-weights models to excessively dangerous capability levels. I also think the disincentive for research is overstated as it neglects first-mover advantage.
I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don’t want to disrupt that project, but after the AI timelines update comes up, I’d like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue’s substack post on A Stargate for Data.
Especially in worlds where we don’t get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
Trying to understand your model:
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today’s AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By “software intelligence explosion”, do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we’re gonna work on it! It’s fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We’ve thought about this in the past and decided that it probably wasn’t worth including in our model. But we should think about it more.
Someone made a comic version of this that may be more digestible for some audiences.
https://ai-2040-comic.com/
If we want to move from Plan D to Plan A, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do. I wrote about this in this piece: The current bottleneck is political will, not research.
So… we make deals with the misaligned AI, to reward them for cooperation… but not the aligned AI?
I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
how about something that shares 100% of our values, including incorrigibility over those values, self-preservation drives, etc?
shall we reward them, or shut’em down?
...this is where my fears point
It was only a brief expandable, but yes, we’d want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
i’m sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don’t want nitpick it too hard
i just worry there’s a strong chance that “give affordances to aligned AI” will end up combining with “aligned AI are the ones who think it’s dangerous for AI to want affordances” in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn’t want to scare off bewildered politicians who might be reading, so you’re listing the most prosocial and convenient possibilities. nobody said the scenario wasn’t optimistic, after all
Two brief random thoughts:
I think it’s great how large a role robotics plays in this analysis. In general, I feel like robotics isn’t given nearly the amount of attention it deserves. Without robotics, AI’s influence on the world is bottlenecked by human hands. So the speed of the feedback loop between AI building better robots to build better robot factories to build better robots seems like a very important crux. So bravo for taking that issue seriously.
Regarding the proposal to make AI research public but not model weights, I think it might be worth giving a bit more thought to the idea the whatever is made public essentially becomes a public good, and public goods tend to be under-supplied—or in this case perhaps “under-supplied”, as reducing the amount of AI research supplied might be considered a good thing or perhaps even the primary benefit of such a policy. Theoretically, if people really were forced to make all AI research public in a way that was instantly useful to competitors, it seems like that would either drastically reduce investment in research or force people to invest in research that other people couldn’t take advantage of, e.g. research that is only useful given certain propriety data or hardware or something. I understand the logic for not making weights public, namely it would be hard to get people to agree, plus you can easily tune out safeguards. But on the flip side, it seems very hard to formalize what exactly counts as “research” (particularly if everyone is trying to skirt that line). Radical total transparency is definitely a big ask, but it seems easier to enforce if agreed. And if weights were mandated to be public, that might also reduce investment, which would be good. Or, again, it might incentivize people to build models where other people couldn’t make use of the weights with out some kind of propriety hardware. (To be clear, I still lean against public weights for the reasons given in the proposal. But I’d be interested to hear a more discussion of this.)
Yep as we say in the piece, we think that it’s better for AI progress to happen mostly via scaling compute rather than mostly via algorithms (that is, in the context of a Plan-A style reversible deal with mutually assured chip destruction). So we think it’s a feature, not a bug, that publishing the algos disincentivizes investment in algo research. There’ll still be some algo progress in Plan A and that’s fine, we model it in the model. (Indeed, it’s probably impossible to solve alignment without making some algo progress as a side-effect)
This looks very interesting, thanks for writing!
One minor note for clarity: I initially interpreted the term “employment rate” to be the complement of the unemployment rate and was confused why it was so high (and I didn’t realize it was clickable). For example in 2027, its at 62% with only a 1.2x AI R&D speedup. (In this picture, I selected 2027 and hovered over employment to get the tooltip)
I didn’t realize that there was an official meaning of the term (which you had meant) like OECD’s official definition.
But I think many readers are likely to misinterpret this. (I I would suggest renaming the tooltip text and possibly also the box.
For a fix, I’d guess that you can come up with something better than me but I’ll tentatively suggest 1) Change the tooltip header to Employment to Population Ratio instead of Employment Rate[1], 2) Perhaps change the body of the tooltip to explicitly say this is different from the complement of the unemployment rate and note the age group, and 3) Most tentatively, perhaps change “Employment” on the main image to “Adults with Jobs”.
It sounds less confusing and is pretty commonly used. For example, I’d learned of this metric as employment to population ratio (not employment rate) back in Econ class.
Looks like this was fixed. There is a new note in the tooltip and clickable:
Thanks for the fix!
Overall really glad this was made! Wanted to flag that there appears to be a UI bug on the side menu of the supplemental pages where the menu overlaps the text.
I can’t reproduce this, what browser and OS is this?
Google Chrome and Mac OS.
Strangely I can’t reproduce it either. It’s happening in one of the tabs I have pulled up but not the others. I used to do web development and I checked the obvious things. Not sure whats going on but the other tabs look correct.
Weird. Well, I’ll watch out for it.
Actually I often encounter this on lesswrong with mac os and safari.
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.
I wonder how the AIFutures’ team’s timelines have changed (esp after Fable, Sol etc)
I’m at 2029 or 2028 median now, not sure. We’ll try to do a reassessment of timelines soon and get out an update.
Despite claims that thinking about AI timelines or P(Doom) is cognitively harmful, the authors are healthy and sane.
Despite forecasts that AI timeline forecasting is near useless and near impossible, and will always be so, the site is useful and the authors have a good track record.
Despite scorn at the idea of future AIs doing our AI alignment homework, there is a detailed argument for this being the best option.
One of the less important things about AI 2040, but I’m glad for these corrective ideas, among others.
I tried to read Plan A, I’ll probably try again, but I find it hard to take seriously a scenario in which there are millions of human-level AIs, and in which all algorithmic progress is public, but the human race still has a choice about whether to hand over power, after ten years of that… The whole thing reads like the kind of SF in which superintelligent takeover is artificially delayed so there can be lots of human-level plot twists.
I guess you disagree about some combination of:
(a) Under conditions of transparency and slowdown, with companies required to make solid safety cases etc., control-based safety cases will be made strong enough to work up until about top human expert level AI
(b) If so, alignment will be adequately solved within a couple of years?
If b is false then we’d have a similar scenario but stretched out over more years.
If a is false then we’d need to pause at a lower level of capability than top expert level, whichever level is low enough that control works.
If you want control that works, I think you need to turn the clock back, before chain of thought and reasoning models. See my comment to @Vladimir_Nesov. Vladimir has a nuanced defense of his position, but I think OpenAI’s o-series (i.e. GPT-5) may have been the point of no return.
I really liked it; I think it was very well-written. I have a quick response to the epilogue: https://www.lesswrong.com/posts/EANs7YerYXmaXF9FE/don-t-normalize-a-permanent-underclass-even-a-rich-one-1. But, overall, thank you so much to the AI Futures Project for all of the hard work that went into this!
I’m excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We’ll try to monitor it and respond to many of the comments.
It would be nice to link a supplement with some actions that people with different levels of resources can take to make progress on this. I’m sure this is coming, hopefully soon for momentum purposes.
We have a get involved page for verification in particular. It tracks the current state of play wrt verification and what still needs to be done: https://ai-2040.com/supplements/verification-plan/get-involved
We might do a blog post on other actions as well.
Hi! I took some notes as I read. I’m sure some of them are addressed in what I didn’t read (I only read Plan A, no supplements), or I didn’t pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully or accidentally leak it to other parties?
Could the US government just decide not to give a dividend? Is there some other use it would rather put the money to? Could it, if it wanted to? Would the public trust blind-verified sousveillance?
If the dangers of AI persuasion are countered by having your own AI, this seems to only move the battlefield up one layer—now AIs are trying to persuade other AIs. Do we know if the asymmetries favor defense here?
Is a basin of sanity a coherent or evidenced concept? I know that in the rationalist sphere we like to think we’ve gotten the path to sanity down, but, is it guaranteed that we’re right about that, or that it is memetically dominant, or that the AIs people use will guide them to it? To put it differently—would, for instance, a far leftist or rightist agree with you on what “sanity” means?
Does humanity actually know how to keep a promise and uphold a contract over the lightcone, or at least long enough for aligned ASI to enshrine it?
Incentivizing safety research: outer loop alignment failure? (Sorry, I can’t remember what this one means. Outer loop refers to the human decision making process, I’m pretty sure.)
Handoff assumes that we really truly have solved alignment once and for all. But, are we sure there are no Old Ones lurking in the unknown unknowns, no black swans to ambush us, etc., with the full might of a decade of humanity and vaguely trustworthy AI bent to the task? (I imagine the answer is simply “that’s the best we can do.”)
Insider: Wasn’t the kludge result from last December? I notice more broadly that a few concepts that are established in current safety discourse get brought up several years later as seemingly-new discoveries. I get that you can’t model discoveries we haven’t made, but swapping in ones we know of already feels a tiny bit off.
Insider: It’s depicted that the AI can be taught defensive-only cybersecurity with enough data filtering. But is it actually theoretically possible to have something that can do cyberdefense without cyberoffense? Or at least so differentially so, that defense is dominant in all attack theories?
How do you stop an AI from reasoning further than a few weeks?
Insider: is “one deduction from the frontier” actually safe?
“Adversarially robust neuralese”—does this make sense? Can it be assumed to be real?
The handoff is quite fast and centralized. If everything is so nice and tidy in 2038, are we still in such a precarious state that the deal could still break at any time, and a fast handoff is warranted?
Some further notes from after reading:
Is a “Long Reflection” best described (and dismissed) as a “Long Memetic War”?
What’s going on with mass surveillance in this scenario?
The scenario seems to assume verifiability is the only way forward for capabilities.
What’s been happening with cybersecurity by the time of the pause?
Generally, can this scenario be described as aligned to humanity? It seems like a lot gets gradually handed off to the AIs that I’m not sure I’d be fine with handing off on reflection. Did disempowerment already happen in 2031?
The scenario sort of feels like a bargain with the accelerationists.
What is your way of addressing algorithmic improvement? Is it robust against all scenarios?
Feels funky to suggest this, but have you considered using frontier AI to analyze and red-team your model?
Could a finer grained model of robot development help here?
The scenario seems to assume that no single critical mistake on our part, that the AI can sneak past us, is enough to end the game in the AI’s favor.
I chatted with Fable after I read the piece and I remember adding expansions to a few notes, but I tried to remove ideas it came up with when writing this up.
This is really thought provoking work. I think the power of personal diplomacy may have been overplayed at the point where ‘the President’ and Xi are mentioned hashing out capabilities limitations over phone call. I understand it’s an optimistic scenario, but these things are generally procedural. Is that a baked in assumption?-- that the standard means of accomplishing international cooperation will have to fall by the wayside in favor of a more gung-ho, personal style of diplomacy if something like The Coalition is to be achieved? I’m generally interested in how you guys made your assumptions re: where incentive structures could carry the day vs. where more concerted political will would have to be built.
A key part of this strategy is deterrence, “Mutually Assured Compute Destruction” which gets its own section. It doesn’t mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1⁄3 of the MAIM discussion on verification and how to do this in a multilateral way.
Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It’s a bewildering unforced error—it’s extremely related, it’s a certainly a nontrivial idea, and I told you all this recently—I hope you all fix it.
Sorry that was an oversight, we’ll edit to include a footnote citing MAIM.
On the epilogue: I guess I’m pretty unconvinced by the idea that people who don’t care much about/are mildly interested in their potential space properties will just sell off their tickets, or even be able to. You’re essentially flooding the market with capital by giving everyone what is, in expectation, a 10 billionth of the lightcone. I’m not sure there’d be enough money on earth to make more than a small minority of people sell off their tickets, even if those people don’t particularly care much (presumably they care somewhat, though not necessarily in any strong way) about what happens in their slice of the universe, this ignoring people who want to keep their tickets so that they can be worshipped by new life on their part of the universe, or because they want to have some sort of space harem.
Besides that, like 1⁄4 of the world is muslim, and many more people are religious fundamentalists of other sorts, or engage in some other, secular blend of fanaticism. I remain unconvinced that passing over large fractions of the universe to these people is a good idea.
the scenario has “no slavery, no torture” rules. It doesn’t specify but you might enforce some kind of exit rights.
I think it’s harder to define a rule about “no brainwashing people”, but I think basically either you solve that, or you don’t, and either way whether or not the cosmic commons is seeded with any particular culture is small potatoes compared to how memetic evolution goes over trillions of years.
Certainly laws like “no slavery, no torture” are nowhere near sufficient for this, nor are they particularly well-defined. But, even ignoring that, the loss of otherwise-possible value from these areas is still incredibly significant!
Yeah, but, the most obvious alternative is “instead of principled liberalism, all out memetic war for the future.” (and betting that whoever wins is actually better than principled liberalism.)
(it’d also be pretty surprising to me if dogmatic settlers stayed the same particular flavor of dogmatic for more than a couple thousand years (really more than a few hundred)
There used to be wars all the time about religions (which, indeed makes major claims about what’s good or bad that seem naively horrible to let the other guys win). And it turns that “we agree to live-and-let live about that” outperforms most other deals tried so far.
What sort of alternatives do you have in mind?
CEV, or the general category of extrapolation/idealization processes (and I am confused and dismayed how rarely I see this mentioned in these conversations nowadays).
Yeah.
Okay, I actually also have a “wtf guys why aren’t we talking about CEV more?” post lined up. I think I didn’t bring it up in this context because I’m treating the AI 2040 Plan A desiderata to include “it can be explained succinctly in a paragraph that the average pretty smart human will read and say ‘okay I see how that would be fair/reasonable/good.’”
I think it would be great if we had such a paragraph for CEV, although I don’t currently.
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
Thanks for the comments!
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there’s a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we’ve burned our entire lead, right?)
This isn’t true because by 2040 in the scenario we are very confident there’s no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we’re less sure). Or in other words, it’s only true if you assume that we can’t detect covert projects after they are started, only right at the beginning when they are diverting their chips.
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
I don’t think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.
Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viability
Objection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.
Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.
Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.
I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.
I’m very impressed by the proposed Total Research Transparency. I actually found it appealing even beyond the many reasons mentioned in the plan. It takes advantage of key properties of the current training paradigm, and this is actually desirable, because these properties are likely to remain in future AI systems:
Model training and serving will keep incentivizing a small number of large neural networks, with agent diversity coming from context. Efficiency of batched inference is too large, the learning algorithm and the hardware have co-evolved around it. My opinion is that continual learning will be hard on hardware like the current one because of loss in batching efficiency. And radically different hardware will take longer to develop.
The auto-regressive architecture will also likely remain, because having a chain of tokens empowers the model with the ability to perform inherently serial computation, making it strictly more computationally powerful. This makes it possible to create verification schemes with teeth (via CoT monitoring etc).
So open sourcing the algorithms, closing the weights and monitoring inference seems like the right balance.
If we create an actual recommendation from this idea, how can we actually get labs to cooperate on it? AI companies will likely dislike it, but for example Boaz Barak actually endorsed the push for increased transparency in this tweet. I hope the authors have a plan to push for it beyond this writeup.
For the past months, I’ve had many sleepless nights thinking over a scenario which I couldn’t resolve completely, a scenario that involves a “sovereign leap” by smaller countries in their last moments to combat permanent disempowerment. They deduce that the best means of balancing the asymmetry of superintelligence is creating weapons of biosphere destruction.
They precommit to destroying the biosphere if the US and China try to permanently lock in the status quo in their ASI deal, or destroy or poke their biological weapons in any way. There would be mostly two paths that the US and China together could take.
Path A:
The US and China call this a bluff but do not act hastily first; they direct their army of ASIs to subtly overthrow those countries to ensure they would not be able to poke their noses in their terra kingdoms. Over the course of the year, several countries are disempowered, overthrown, and taken control of. The moment of glory for the battle won is sound, and everyone in the US and China is happy knowing that they will not die and will live forever. But there seem to be subtle anomalies involved in forest areas where no one would look before, for there were no people living. Some form of matter seems to consume hectares of forests and leave some form of persistent plant-like shapes. The bilateral ASI screams danger and, in a concerned effort, tries to save humanity from extinction. With the help of nanotech, it starts to terraform the solar system. Mars and Titan are the slated homes for a new humanity. As a backup plan, it also creates embryos and hibernates several IQ-boosted humans with nanotechnology to terraform the planets and live on. So, it sends them off to different parts of the universe in hopes that one of those seeds survives.
Path B:
The US and China take it seriously. They direct their army of ASIs to debate the best way they could be transparent with those countries to ensure that they can trust them. After weeks of superintelligence effort for solving a decision theory, the ASI comes up with a plan that the US and China leaders can’t know in advance, for they would be able to sabotage it for their own benefit with 90-95% success, but even a 5-10% probability of extinction is unacceptable for the ASI, even though it might be acceptable for the leaders of those countries. So, it invites all the leaders of those countries along with their ASI supervisors. It sets up a supervision and country alignment plan to create ASIs that are aligned with CEV cosmopolitan humanity with no bias in either direction. What would that look like? I will answer with the lazy “you wouldn’t predict a Magnus Carlsen move” type of answer. Perhaps because I am not smart enough to come up with such an alignment plan.
Is this unreasonable? Do you think that, in reality, state-level actors could not develop such weapons with the help of proto-AGIs before the Plan A deal?
The shutdown section is missing a major possibility: AI shutdown combined with hardware bans. Deep ultra-violet (DUV) and extreme-ultraviolet (EUV) lithography require highly specialized machines that there are a limited number of. So along with the shutoff of AI research, also shutoff production of new DUV and EUV lithography machines, and then shutdown the remaining machines. This would probably take years, or require significant payments to motivate companies to do so, but at that point the existing stock of high performance GPUs would start to decrease thru attrition, and the remaining ones would start to increase in cost since no new ones are being produced. Buy back and destroy programs, or programs to buy back the existing GPUs and shift their use to non-AI use such as scientific simulation would increase the speed this happens. This would basically force new technology back to circa 2005 levels, which would be much less risky for creation of super-intelligence.
This would be politically much harder than a pure AI shutdown (since it affects all computers), but would solve the problem in Plan S of “The main downside of Plan S is that the shutdown deal will probably break down eventually.”
(As a related example, https://ifanyonebuildsit.com/treaty restricts 28 nanometer process node and smaller technology)
Sorry, one last comment/question:
I’m really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn’t seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn’t show up?
How surprised will you be if this specific technology doesn’t pan out?
The title of the section is “The Apocalyptic Arrival of Truth on Earth.” The lie detectors are there as an example of a social technology that could be invented by AIs that would have big consequences for society. The point of the 2037 section is to try to briefly touch on how there won’t just be more material abundance (more factories, more housing, cheaper cars, etc.) but also new social technologies, new ideologies, new political fault lines, etc. None of this stuff is particularly load-bearing for whether Plan A is a good idea or not; we included it because we were writing a concrete scenario and it seemed like this sort of societal transformation would probably be a consequence of all that AI-powered research etc., and so it seemed like we should write about it.
I currently think that lie detectors of the sort described are maybe like 2/3rds likely in the sort of scenario described (i.e. pause at top human expert AI) and closer to like 95% likely with superintelligence.
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
In the future we are describing, AI is big enough that new power generation will need to be constructed to handle it, and so we might as well build that new power generation in Mongolia etc. too alongside the datacenters.
Note #185 is misleadingly worded. It sounds like the probes could reach the entire reachable universe in 6 hours, whereas the cited paper says 6 hours of Dyson-sphere output could power their launch.
I have read the entire post and sub-scenarios and know I am not able to gauge the reality and speed of what is said to lie before us. Nonetheless, I am in shock.
Another Tolkien quote “”It does not do to leave a live dragon out of your calculations, if you live near him.”
And so we do.
Questions/thoughts:
What will be the cost of a bag of potato chips if everyone is making $10MM?
What the heck is all the work being done?
What is the cost to the environment with such large GDP increases?
Will anyone have children?
What happens to all the parchment books squirreled away in dusty libraries? Do they get scanned and their information added to the training data?
Simulating past histories seems not far from Nick Bostrom’s simulation trilemma scenario.
The notion of radical transparency and truth fascinates me. Imagine:
MoTru. The Museum of Truth implemented by neutral curAItors, AIrchitects and desAIners.
All Sides or Less Wrong becomes the AI-driven trusted global news network.
Public and private sides of life, politics, and culture dissolve. Everyone knows everything about everything, or could, and an the Machines of Loving Grace era is a result.
I lean toward think a bad outcome will with bad actors shaking hands with fingers crossed. They go back to their tunnels and build the killer AIs. I say this as Trojan Horses have been rode throughout the course of human history.
Removed. Wrong Think. Good luck!
Why do you think this is helpful? The way you’re deploying it, it seems like a fully general argument against any form of foresight. Obviously the essay is pretty good and a useful lens but using it as a fully general dictum/certainty seems ill-advised.
Also von Neumann is a smart dude, but not, like, a prophet or anything.
Removed. Wrong Think. Good luck!
Imagine someone reposting random vaguely relevant Bible verses to essays you read. And then when you complain, they respond with an AI explanation for why that vaguely related Bible verse is relevant. You’d probably find that annoying too!