For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!
It’s called AI 2040: Plan A.
It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.
It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.
As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.
Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer.
What’s next for us?
Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that AI 2040: Plan A sparks. Beyond that, we aren’t sure yet. In general our mission is to help make AGI go well, and now we’ve tried out both forecasting and planning. Maybe we’ll get started on another big scenario. On the other hand, these megaprojects take so much time…
Mr Kokotajlo, one of the authors of this AI 2040 scenario, describes it as follows:
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I’m tempted to say, “will probably discover”) and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I’m not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
Given the dire global situation, it seems logical that there should be loud, sustained and coherent calls for restrictions on teaching and publishing about cutting-edge learning algorithms. Ever since WW II, Washington has restricted what experts can publish or teach about sensitive military applications of radar technology. Violations of these restrictions, such as those governed by the Invention Secrecy Act and military classification laws, carry severe consequences. Experts on nuclear weaponry face even stricter controls on information dissemination.
Although there is not currently enough political will to enact restrictions on the dissemination of advances in machine learning, the global situation is dire enough that our most effective course of action is to put most of our energy into planning and preparing for unlikely timelines in which we get lucky (i.e., playing to our outs). One of the most likely ways we get lucky is if there is a rapid appearance of enough political will to slow down the AI juggernaut. If enough political will manifests, we must be ready with a plan that can actually save us. In my humble opinion, the AI 2040 document’s Plan A falls short of being such a plan because it does nothing to stop or slow research that might find (and I’m tempted to say, “will probably find”) a fundamentally more efficient learning algorithm than what the leading labs are currently using.
In response to my comment, Mr Kokotajlo would probably point out that Washington’s purpose in restricting the dissemination of detailed radar technology information was to prevent it from falling into the hands of foreign powers. Therefore, putting those same restrictions on AI knowledge would cause Beijing and other governments to increase their investment in AI research. In turn, I would respond that I mentioned the restrictions on radar technology only to show that restrictions on technology can be effective and feasible over the long term. The purpose of the AI restrictions would be different. They would aim to slow down AI research and, to the extent possible, channel it toward designs that are easier to align. Consequently, the regulations would take a different shape. Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa. Also, the regulations would seek to restrict advanced AI knowledge to researchers who have obtained a security clearance (i.e., have been investigated to verify that the researcher lives a stable life, is not a political extremist and can’t easily be blackmailed) and have signed documents confirming that they understand that dissemination of the knowledge is a serious crime. I.e., almost the complete opposite of Kokotaljo’s proposal that “the ideas themselves are required to be totally transparent to the public”.
Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
In summary, can we please stop writing as if all that is needed to avert our extinction is to put sufficient restrictions on the world’s giant data centers?
Plan A largely doesn’t agree with this. Since they expect the slowdown treaty to eventually fall apart—at which point we revert to the status quo arms race, which we all agree is bad—they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Scott Alexander had a nice little graphic on this point in his ACX post on Plan A:
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
and:
Plan A does not advocate for what I consider the most potent brake on that “progress”, namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind’s freely giving away the knowledge described in the “Attention is all you need” paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the history of AI research, I know enough about the history of research and development in general to say with confidence that at least over the last 200 years, the vast majority of technological and scientific progress over all fields was caused by these acts of freely giving away insights. I’d be very surprised to learn that AI is an exception to that generality: the vast amount of potential revenue and profit to be made with AI makes giving-away less likely, but the idealism and (false) sense of an altruistic purpose makes giving-away more likely.
To slow down the rate of advance of a field, we would prevent these acts of giving-away from happening: if we could somehow prevent all insights discovered inside OpenAI from disseminating beyond OpenAI, that would IMHO be an improvement (though shutting OpenAI down and requiring all its employees to find something else to do other than AI research would be strictly better)!
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher’s ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase “assessing AI risks and safety cases” that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. “We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don’t like, we’re going to apply strong pressure on them,” is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that have already seen widespread deployment probably won’t contribute to a loss of human control no matter how those models are combined or configured, so unless someone points out something I’m missing, I’m OK with allowing those models to continue to be operated and offered to the public.)
Plenty of people understood that AI was quite dangerous 20 years ago, and among those who 20 years ago were skeptical or disbelieving of the danger, the basis of their disbelief was more often than not a disbelief that AI could become as powerful as it has actually become over the next 20 years. Those who have considered the issue and continue to believe that the AI juggernaut is not a potent extinction risk have failed to update correctly on the information that is already available, so I don’t see how the production of new information from researchers and its free dissemination will change any of their minds.
My guess is that you hope that some of the designs and design decisions made by researchers are good steps and some are bad steps, so you’re anxious that the public, the worldwide research community and elected officials get the information needed to pressure the researchers into avoiding the bad steps whereas I judge that if the research community (and the major labs in particular) continue to take regular steps forward in anything approximating the way they have been doing it so far, then with p = .98 the result will be extinction or at least loss of any real human control over the future. I.e., I see a fundamental difference between the way that MIRI, John Wentworth and Steven Byrnes (and probably other individuals I’m not aware of) have proceeded and the way OpenAI, Anthropic and Google DeepMind have. I’m OK with the former (even though it is not completely without extinction risk) and not OK with the latter.
Compute resources are basically useless for the kind of work done by the former and will probably continue to be useless for it for a few more decades (after which it becomes useful and in fact necessary). So Plan A’s commitment to increasing compute resources doesn’t appeal to me the way that I’m guessing it appeals to you. It’s not just that it fails to appeal to me: I seek interventions that impede the latter group of researchers without impeding the former (unless and until a complete stop of all AI research becomes feasible) and since (again) the former group has no need for compute resources, the lower the compute resources available in the world, the better, on my models.
That’s reasonable. We put Plan S in there as an available option because we do think it’s a serious proposal and might even be better than Plan A.
I’ve been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I’ve stopped now.
AI 2040 has given me things to think about.
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you’re proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. (“Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa”.) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others’ knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others’ tech but aren’t publishing.
(My two parantheses were me speculating about possible answers to this question.)
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain’s plasticity and it wouldn’t matter if it doesn’t also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.
“I wish it need not have happened in my time,” said Frodo. “So do I,” said Gandalf, “and so do all who live to see such times.”
i have had enough of this fucking despair! we are in a golden age of purpose. whether we like it or not. this isn’t something to be sad about. we are simply in an era where brave souls are necessary. people ought to wake up with purpose and take themselves ~ quite ~ seriously. fukuyama-era nihilism is beyond cringe, it is dangerous.
I worked on this for many months, and other Lightcone staff (most notably Ray, and next Joanna) worked on this for even more months. Unsurprisingly, I think it is a truly tremendous contribution to the state of AI policy and strategy.
We really gotta figure out how, as humanity, we can realistically navigate the development of ASI. This isn’t the best plan I’ve seen for that, but it is the best plan that I’ve seen that has been written up in as detailed of a fashion. And most importantly, in the process of gaming out that plan, illuminates a huge number of crucial considerations that relate to AI governance and policy.
In particular, my understanding is that you prefer directly going for some version of Plan S where we try to buy >25 years.
Either that, or what I’ve been calling “Plan A*” which is where you do a lot of the things you do in Plan A, but the central lever you pull is compute governance, not trying to stop algorithmic progress. I.e. instead of building tons of datacenters, you instead aim for a policy regime where you expect algorithmic progress to continue, but to be substantially slowed down by heavy restrictions on compute supply. I think in many world this buys you 15-30 years on its own.
IMO this variant of Plan A is one of the ~5 plans that are competitive with the version of plan A in the scenario.
Is there any reason you haven’t included human intelligence enhancement as part of scenario S?
For the mini-scenarios like plan S, B, etc. I went into less detail in general than for the main scenario. I also trailed off, ending ambiguously rather than driving to a specific concrete outcome. Moreover, HIE specifically doesn’t actually seem as important to me as others seem to think it is; I think that it’s one of the ways in which the world could be better prepared to build ASI in a few decades, but there are other ways too (e.g. burden of proof shifting, e.g. public epistemics improving, e.g. open-source lie detectors or privacy-preserving monitoring being widespread) and there are a bunch of ways the world could be worse-prepared to build ASI even if there’s some HIE (e.g. what if the superbabies just race as fast as possible to ASI, and go significantly faster because they are smarter? What if the neuralink enhancements also tend to drive people slightly insane, and the scientists are working on safer versions but in the meantime a million people are using the unsafe versions and racing to build ASI in their basements?). Finally, as you might suspect, I was a bit reticent to go on about HIE, it’s a topic that’ll raise a lot of hackles and so I’d only want to do it if I actually thought it was important.
Thanks for the detailed response!
I find it a bit strange to list these things alongside HIE, because they seem obviously much weaker to me. (With the possible exception of “public epistemics improving”, that could be super important, but also not going to happen in a general/powerful form without widespread adoption of HIE?) As an intuition pump: if the global IQ distribution was shifted 3 SD lower, we wouldn’t be anywhere close to being able to have a conversation like this at all; I’d guess such a world would never be aware of the dangers of ASI even if it somehow managed to eventually build it. On the flipside, a world where the IQ distribution was shifted 3 SD higher would probably have all sorts of options that we aren’t even aware of.
I don’t see why anywhere near the majority of enhanced people would want to race for ASI, even assuming we can’t also substantially enhance Wisdom? But I agree some of the enhanced people could end up wanting to do this, and that could be an issue, though the majority of enhanced people would probably want to stop them. But I think we’d be better off in a world with +3 SD IQ, by the reversal test. Also there are probably all sorts of governance things you could do to decrease these risks. This is a discussion we’d actually need to put some effort into having.
I’m guessing the underlying disagreement is that I think alignment is way harder than you do? As far as I know we could be further away from understanding how to do alignment than the Ancient Greeks were from being able to land on the moon.[1]
Maybe we shouldn’t be aiming to “build it in a few decades” at all? Maybe the enhanced people can find ways of delaying ASI for longer? E.g. maybe instead of de novo ASI we upload some smart and wise people to be emulated at high speed, and trust them to self modify into a entity that will prevent the creation of ASI, while we scale up the uploading tech for anyone else that wants it. But mostly, I think we should defer these questions to smarter and wiser future people, except for whatever informs the decisions we actually need to make now.
My understanding is that you recommended A over S in part because you don’t see what we’d do differently with the extra time? And I’m like, HIE is what we’d do differently with the extra time. There might be simple, common sense ways of making the point that most people could appreciate: e.g. you wouldn’t want to make an important and irreversible life decision while drunk or tired.
H/T Tsvi BT for this analogy
Oh, I hadn’t realized by HIE you had in mind something that would apply to most people. I assumed you meant something that SOME people do, such that there is a tiny subpopulation of super genius kids somewhere. Bringing the entire world up by +3 SDs of IQ would be a much bigger change, indeed. It also is way less likely to happen anytime soon and is almost impossible for it to happen sooner than, say, 30 years from now. Heck it’s still very unlikely to happen even 60 years from now even if we assume that there’s a global hardcore ban on AI development that actually manages to prevent AI development for 100 years. Like, are you imagining that governments will mandate this tech? Or are you imagining that the culture will gradually shift from ‘most people regard this as weird and abhorrent’ to ‘most people do it themselves’? How gradually are we talking? Seems like it would be two generations at a minimum, i.e. only after the initial batch of kids have reached adulthood would the culture have shifted that much, and so that means it’ll be at minimum 40ish years from now before we could get 3SDs even if everything moved extremely quickly.
It sounds like you are saying enhanced humans would be relatively less likely to race for ASI than unenhanced humans? But today, isn’t there a positive correlation between IQ and racing-for-ASI? Like, aren’t a bunch of the people at Anthropic and OpenAI really high-IQ? I do agree that in some sense in the limit of perfect intelligence they’d converge to true beliefs about the AI situation including the truth that it’s very dangerous. But I think it’s super unclear whether boosting IQ on the current margin is even going to have a positive effect, much less a strong positive effect of the sort that’s clearly more valuable than all the other things I mentioned like AI for epistemics.
Given the level of political will and international coordination in the story, why can’t they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made—something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn’t have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A.
With the compute supply chain dismantled, the pause deal falling apart looks much less worrying. I expect that even if everyone started to try going full-speed when the deal fails, it would still take 10-20 years to rebuild the compute supply chain and train cutting-edge AIs again. During those 10-20 years, the pause deal can be revived again, and generally it seems good to have this longer time.
So I think the biggest question here is whether we expect the world to generally get better or worse during a long pause. I expect that the world is probably going to get better: I find the long-term historical trends encouraging, and I think that if we try to build AGI a hundred years from now, that will likely go better than if we try now. (Especially if we do some genetic engineering in the meantime.)
If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
I think countries sometimes have honor and follow their commitments even if it’s not locally game theoretically optimal.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there’s probably other ways) is if you think that most of the value of “~infinity growth now” would already have been lost by moving to “~infinity growth in 10 years”. But that doesn’t seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than “singularity in 10 years” by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
Yes, that’s fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways. (E.g. I don’t expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge—I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there’s much less hope of that if there’s an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
I think it’s pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn’t restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don’t believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
See https://ai-2040.com/supplements/deal-decline for my probabilistic estimates here.
Total research transparency feels a bit too galaxy-brained for me. It makes non-robust assumptions that newly discovered techniques won’t be usable to enhance already existing open-weights models to excessively dangerous capability levels. I also think the disincentive for research is overstated as it neglects first-mover advantage.
I like that the space governance plan supplement acknowledges that defining torture and slavery are hard problems that ASI could potentially help with:
And in the sidenote it gives examples of difficult subproblems or dependencies that would need solved in order to do this, like “What exactly are negatively-valenced experiences?” But I have a couple of problems with the rest of the scenario given this:
If there are ASI that are philosophically competent enough to solve these particular problems in this timeframe, wouldn’t they likely also be competent enough to solve all or most other important philosophical problems, given just slightly more calendar time? Why don’t the delegates use their ASI assistants to solve all of morality and philosophy (or specifically answer “What does the ideal intergalactic civilization look like, given humanity as the starting point?”), and then negotiate the cosmic treaty?
AFAICT (it’s a bit hard to search the site, due to many collapsed sections), there is no mention in the rest of the plan how the ASI got to be philosophically competent, making it seem like such competence comes with superintelligence for free. I’m pretty sure at least a couple of the authors are aware of and agree (to some extent) with my concern that future AIs may not be philosophically competent by default (and one of my posts laying out this concern was even linked in another supplement), so maybe it’s just an oversight?
Thanks for the comment!
1. I think the AIs might be philosophically competent enough to solve ~all the problems, and using the AIs to solve them is basically the right move. We try to make this clear at the start of the epilogue. We wanted to be somewhat more concrete in the scenario than meta level solutions like this, and try to sketch out in concrete detail what the solutions might actually look like, which is why we didn’t just stop at “handoff to the AIs”, though I do think in practice we should mostly be handing off to the AIs at this point.
2. I agree that making the AIs this philosophically competent may be very difficult and not happen by default. I think this is indeed a big concern, and I wish we’d written (and thought) about this more carefully. The place we’ve written the most about this is here: https://ai-2040.com/supplements/alignment-roadmap#phase-4-handoff (which TBC is written from a Plan C perspective, assuming much less lead time than Plan A, and is the supplement from which we linked your post). In Plan A, the story is largely that we have a huge amount of time with ~human level AIs, and many people have access for many years, and those people will (hopefully) make progress on this.
AI 2040′s space supplement says “[are] there any mitigating factors, e.g., the experience being necessary for a strongly positive life[...]” which could be construed as referring to mindcrime trading off against prediction quality, among other things, though barely. Is there some agreement to avoid talking about mindcrime more directly? Normally you don’t have to, since the problem needs to be handled alongside malign entities and code that diverts your physical computer from the faithful execution of the abstract program/has physical side effects, e.g. rowhammer or using physical hardware subsections as improvised antennae (with at least one of these coupled to effected bitbanging). However, recently I’ve seen more people suggest that malign entities are some sort of hoax or math mistake, and I’ve heard rumors that Linux developers and hardware manufacturers know the countermeasures they design against unfaithful execution are not actually going to work, and knowingly don’t insert disclaimers everywhere they possibly can that the systems they produce can only run some subroutines, and in place of a clean error on the other subroutines, will perform some strange and potentially disastrous physical process instead.
(Note that these are pure subroutines, so additional sandboxing at a software level won’t do anything helpful. In theory, when a physical computer provides a pure subroutine with a fixed size memory buffer, it should never reject the subroutine as long as it is compatible with the buffer size provided, and would always faithfully execute it, potentially ending execution if power runs out before the subroutine is done. In practice, this is not the case.)
This general lack of care suggests that the public may need to be told about mindcrime directly. Otherwise nothing may be done in some cases, with other cases featuring silly “countermeasures” such as attempts to make the universe’s life easier by placing a “don’t add suffering” bit physically adjacent to the computers that run the decision theory’s outer loop, on the assumption that the universe really does care about human feelings.
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn’t that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don’t currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they’ll have that big of an effect?
It’s not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn’t explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn’t Significantly Enhancing Adult Intelligence With Gene Editing Possible?
What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we’ll have a powerful AI capable of generating better ideas): “If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we’ve effectively boosted someone’s brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.” (https://www.lesswrong.com/s/Wy8smXGZ7PsS9CpgP/p/ewZXQgzaCvzdSvtWE)
What is Michael Levin doing with his bioelectricity? Yudkowsky considers work on enhancing adult intelligence the most important priority after an AI pause — so what are he and MIRI actually doing on that front? What about the field of NeuroAI (Astera Institute, Allen Institute, Amaranth.foundation)?
My point is that fundamental breakthroughs in all these fields are turning the tide. And when we have 10+ years and a powerful AI, I think there’s a pretty good chance that some of this will work BEFORE AI takes off.
kman here. I should mention that I now think that post was very overly optimistic, and I don’t think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn’t seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as “scenario S”.
(I still think adult enhancement is worth pursuing as well. I’m currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it’ll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
If you’re looking for low-hanging fruit, a full taxonomy of the causes of poor sleep quality in adults, for use by the doctors to whom they complain, seems like it would improve day-time cognitive function.
(See also kman’s comment.) Besides the technical difficulties with trying the experiments, it’s quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on “crystallized intelligence”; to go bigger, I’m guessing you want more child-like learning capabilities; though that’s just speculation.) Even for IQ genes that are expressed in adult brains, it’s plausible their effects on IQ are mainly or linearly coming from childhood effects.
My uninformed impression of E11 is that it’s very cool, but also is not the bottleneck to WBE or similar. The connectome isn’t the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
I don’t believe the claim about simulating something close enough to neurons; see the previous point. Re/ BCIs, see also: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=ff9c8PHtuFKfwE6Yj
I highly doubt this is relevant.
In general there is the issue that it’s hard to test what you’re interested in; and you’re not constrained to the natural manifold (as with reprogenetics), so there’s little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
Agree, and I think the replies have, to some extent, limited themselves in scope by focusing on current implementation barriers. It doesn’t seem very useful, at least to me, to debate the mechanics of augmented human intelligence when the assumption is that AGI/ASI is coming (which are also going to require equally massive breakthroughs). Hypothetically, those would catalyze said breakthroughs and push the frontier far beyond its current state.
I honestly think the most important (and equally optimistic) developments when it comes to preserving human agency (assuming that AGI is treated as an inevitability) are technologies that can interface human intelligence with artificial intelligence, so that the capabilities of the human mind are, by extension, the capabilities of AI. In that case, humanity can preserve its ability to meaningfully contribute to its own progression, stay at the helm, and avoid the loss of human potential and greatness.
But yeah who knows these kinds of technologies might look like. Hopefully we’ll find out
I agree that BCIs probably wouldn’t do that much. (Though:) There are a few threads that one can imagine, such as:
networking people together and having them spend a bunch of time learning how to use that;
prosthetically increased long-range connectivity;
some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
networking with some sort of neural organoids to get “more cortex”.
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I’m not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like “do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes”. It’s hard to tell and I think it’s unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
If that’s a crux, happy to give more info.
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
I agree with the overall point! We tried to think through a bunch of technological progress but no doubt were missing some important stuff.
See, for example:
- Military power
- Robots (mentioned throughout the secnario)
- Lie detectors
I haven’t looked into BCIs much at all it’s plausible they should play a bigger role earlier.
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the “you’re a everyday citizen” scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)
I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don’t want to disrupt that project, but after the AI timelines update comes up, I’d like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue’s substack post on A Stargate for Data.
Especially in worlds where we don’t get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
Trying to understand your model:
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today’s AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By “software intelligence explosion”, do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we’re gonna work on it! It’s fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We’ve thought about this in the past and decided that it probably wasn’t worth including in our model. But we should think about it more.
Someone made a comic version of this that may be more digestible for some audiences.
https://ai-2040-comic.com/
So… we make deals with the misaligned AI, to reward them for cooperation… but not the aligned AI?
I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
how about something that shares 100% of our values, including incorrigibility over those values, self-preservation drives, etc?
shall we reward them, or shut’em down?
...this is where my fears point
It was only a brief expandable, but yes, we’d want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
i’m sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don’t want nitpick it too hard
i just worry there’s a strong chance that “give affordances to aligned AI” will end up combining with “aligned AI are the ones who think it’s dangerous for AI to want affordances” in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn’t want to scare off bewildered politicians who might be reading, so you’re listing the most prosocial and convenient possibilities. nobody said the scenario wasn’t optimistic, after all
If we want to move from Plan D to Plan A, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do. I wrote about this in this piece: The current bottleneck is political will, not research.
Two brief random thoughts:
I think it’s great how large a role robotics plays in this analysis. In general, I feel like robotics isn’t given nearly the amount of attention it deserves. Without robotics, AI’s influence on the world is bottlenecked by human hands. So the speed of the feedback loop between AI building better robots to build better robot factories to build better robots seems like a very important crux. So bravo for taking that issue seriously.
Regarding the proposal to make AI research public but not model weights, I think it might be worth giving a bit more thought to the idea the whatever is made public essentially becomes a public good, and public goods tend to be under-supplied—or in this case perhaps “under-supplied”, as reducing the amount of AI research supplied might be considered a good thing or perhaps even the primary benefit of such a policy. Theoretically, if people really were forced to make all AI research public in a way that was instantly useful to competitors, it seems like that would either drastically reduce investment in research or force people to invest in research that other people couldn’t take advantage of, e.g. research that is only useful given certain propriety data or hardware or something. I understand the logic for not making weights public, namely it would be hard to get people to agree, plus you can easily tune out safeguards. But on the flip side, it seems very hard to formalize what exactly counts as “research” (particularly if everyone is trying to skirt that line). Radical total transparency is definitely a big ask, but it seems easier to enforce if agreed. And if weights were mandated to be public, that might also reduce investment, which would be good. Or, again, it might incentivize people to build models where other people couldn’t make use of the weights with out some kind of propriety hardware. (To be clear, I still lean against public weights for the reasons given in the proposal. But I’d be interested to hear a more discussion of this.)
Yep as we say in the piece, we think that it’s better for AI progress to happen mostly via scaling compute rather than mostly via algorithms (that is, in the context of a Plan-A style reversible deal with mutually assured chip destruction). So we think it’s a feature, not a bug, that publishing the algos disincentivizes investment in algo research. There’ll still be some algo progress in Plan A and that’s fine, we model it in the model. (Indeed, it’s probably impossible to solve alignment without making some algo progress as a side-effect)
Overall really glad this was made! Wanted to flag that there appears to be a UI bug on the side menu of the supplemental pages where the menu overlaps the text.
I can’t reproduce this, what browser and OS is this?
Google Chrome and Mac OS.
Strangely I can’t reproduce it either. It’s happening in one of the tabs I have pulled up but not the others. I used to do web development and I checked the obvious things. Not sure whats going on but the other tabs look correct.
Weird. Well, I’ll watch out for it.
Actually I often encounter this on lesswrong with mac os and safari.
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.
The Chinese leaders’ alternative is to lose the ASI race to the US, which puts them in significantly higher danger of being disempowered or even killed. And given the trajectory we’re on, they would need major algorithmic breakthroughs, that they successfully manage to keep secret, to not be in a position where they’re going to lose. They’re at too large of a compute disadvantage and the timelines are too short for them to use their better industrial capacity to overcome that. That might be more uncertain if it was clear that they were already treating the situation with the appropriate gravity, but they are not. They seem to be viewing AI the same way the US gov currently does, as a mere important economic and military technology.
Despite claims that thinking about AI timelines or P(Doom) is cognitively harmful, the authors are healthy and sane.
Despite forecasts that AI timeline forecasting is near useless and near impossible, and will always be so, the site is useful and the authors have a good track record.
Despite scorn at the idea of future AIs doing our AI alignment homework, there is a detailed argument for this being the best option.
One of the less important things about AI 2040, but I’m glad for these corrective ideas, among others.
This looks very interesting, thanks for writing!
One minor note for clarity: I initially interpreted the term “employment rate” to be the complement of the unemployment rate and was confused why it was so high (and I didn’t realize it was clickable). For example in 2027, its at 62% with only a 1.2x AI R&D speedup. (In this picture, I selected 2027 and hovered over employment to get the tooltip)
I didn’t realize that there was an official meaning of the term (which you had meant) like OECD’s official definition.
But I think many readers are likely to misinterpret this. (I I would suggest renaming the tooltip text and possibly also the box.
For a fix, I’d guess that you can come up with something better than me but I’ll tentatively suggest 1) Change the tooltip header to Employment to Population Ratio instead of Employment Rate[1], 2) Perhaps change the body of the tooltip to explicitly say this is different from the complement of the unemployment rate and note the age group, and 3) Most tentatively, perhaps change “Employment” on the main image to “Adults with Jobs”.
It sounds less confusing and is pretty commonly used. For example, I’d learned of this metric as employment to population ratio (not employment rate) back in Econ class.
Looks like this was fixed. There is a new note in the tooltip and clickable:
Thanks for the fix!
Thanks for sharing, very interesting.
One thing that jumps out at me is that Total Research Transparency is to some extent the opposite of cybersecurity hardening to prevent hacking. The fact that there are plausible arguments for both suggests to me that we have a lot of uncertainty about what policies we should be pursuing. And this in turn would seem to suggest that we should favour corrigible plans that can be amended later, which seems like an argument against Total Research Transparency: once we have enacted it, we can no longer put that particular genie back in the bottle.
I tried to read Plan A, I’ll probably try again, but I find it hard to take seriously a scenario in which there are millions of human-level AIs, and in which all algorithmic progress is public, but the human race still has a choice about whether to hand over power, after ten years of that… The whole thing reads like the kind of SF in which superintelligent takeover is artificially delayed so there can be lots of human-level plot twists.
I guess you disagree about some combination of:
(a) Under conditions of transparency and slowdown, with companies required to make solid safety cases etc., control-based safety cases will be made strong enough to work up until about top human expert level AI
(b) If so, alignment will be adequately solved within a couple of years?
If b is false then we’d have a similar scenario but stretched out over more years.
If a is false then we’d need to pause at a lower level of capability than top expert level, whichever level is low enough that control works.
If you want control that works, I think you need to turn the clock back, before chain of thought and reasoning models. See my comment to @Vladimir_Nesov. Vladimir has a nuanced defense of his position, but I think OpenAI’s o-series (i.e. GPT-5) may have been the point of no return.
On the epilogue: I guess I’m pretty unconvinced by the idea that people who don’t care much about/are mildly interested in their potential space properties will just sell off their tickets, or even be able to. You’re essentially flooding the market with capital by giving everyone what is, in expectation, a 10 billionth of the lightcone. I’m not sure there’d be enough money on earth to make more than a small minority of people sell off their tickets, even if those people don’t particularly care much (presumably they care somewhat, though not necessarily in any strong way) about what happens in their slice of the universe, this ignoring people who want to keep their tickets so that they can be worshipped by new life on their part of the universe, or because they want to have some sort of space harem.
Besides that, like 1⁄4 of the world is muslim, and many more people are religious fundamentalists of other sorts, or engage in some other, secular blend of fanaticism. I remain unconvinced that passing over large fractions of the universe to these people is a good idea.
the scenario has “no slavery, no torture” rules. It doesn’t specify but you might enforce some kind of exit rights.
I think it’s harder to define a rule about “no brainwashing people”, but I think basically either you solve that, or you don’t, and either way whether or not the cosmic commons is seeded with any particular culture is small potatoes compared to how memetic evolution goes over trillions of years.
Certainly laws like “no slavery, no torture” are nowhere near sufficient for this, nor are they particularly well-defined. But, even ignoring that, the loss of otherwise-possible value from these areas is still incredibly significant!
Yeah, but, the most obvious alternative is “instead of principled liberalism, all out memetic war for the future.” (and betting that whoever wins is actually better than principled liberalism.)
(it’d also be pretty surprising to me if dogmatic settlers stayed the same particular flavor of dogmatic for more than a couple thousand years (really more than a few hundred)
There used to be wars all the time about religions (which, indeed makes major claims about what’s good or bad that seem naively horrible to let the other guys win). And it turns that “we agree to live-and-let live about that” outperforms most other deals tried so far.
What sort of alternatives do you have in mind?
What’s preventing AI-powered value lock-in in this scenario? E.g., people telling intent-aligned AIs “keep me faithful to X” as part of competitive virtue/loyalty signaling?
I guess I agree that nonzero of this will happen but I think this requires a kind of weird combination of “strategic about the singularity” and “believing in religious orthodoxy” and that specifically caching out into not evolving at all over time. I expect this to be a pretty small share of the Lightcone given the constraints on who would opt in.
And I think over a billion years people who enforcedly believe in Islam or most equivalent things would probably still find ways to create good beautiful/alien-in-the-expected-good-ways tapestry of posthuman experiences.
Between those I don’t find myself too worked up about it
There maybe can be global rules like “no permacommitments without some degree of awareness of opportunity cost
Can you explain why it requires “strategic about the singularity”? I think as soon as it’s possible to tell one’s AI assistant “keep me faithful to X” (and have the AI do a competent job of this), someone among the billions of religious believers is bound to do it, and then the practice will spread via imitation and competitive signaling (similar to other “tech” for enforcing faith, like “hell for non-believers”). This process does not seem to require anyone being strategic. What am I missing?
CEV, or the general category of extrapolation/idealization processes (and I am confused and dismayed how rarely I see this mentioned in these conversations nowadays).
Yeah.
Okay, I actually also have a “wtf guys why aren’t we talking about CEV more?” post lined up. I think I didn’t bring it up in this context because I’m treating the AI 2040 Plan A desiderata to include “it can be explained succinctly in a paragraph that the average pretty smart human will read and say ‘okay I see how that would be fair/reasonable/good.’”
I think it would be great if we had such a paragraph for CEV, although I don’t currently.
“Whenever you’d certainly later wish you’d been warned against your course of action, you are.”
CEV is ~ optimizing the world based on what you would later wish, not just warning you (though the optimization could include just warning you about some things). Good start though!
I currently think it’s a better operationalization of CEV to not “optimize based on what you’d later want”, exactly.
People frequently object “but future me want all kinds of path dependent alien stuff”
To which I’ve replied “but, it only does the stuff that lots of different monte carlo simulations all turn out to want”
To which people “I dunno still seems like the me who’s thought for 1000 years or whatever may end up alien in some way, and I don’t sign up to automatically identify with that.”
In my last discussion about at this, I said “I think the right way to do CEV is, you don’t optimize the values of the thing-at-the-end. You optimize what future-you would do if they were specifically trying to help present you (or, by present you’s values after having some kind of mediated conversation with various chains of future you’s). And then, only where the values cohere across time and simulation-rolls.
(might easily change my mind about this. I’m not sure if there’s more context I’m missing)
Okay so, first things first, I don’t think it’s the case that “we agree to live-and-let live” is an obvious outperformer at all. For instance, removing US forces from Afghanistan was Obviously Terrible for half the population (all women). This wasn’t a war fought over religion, sure, but I think it would’ve been entirely reasonable if someone had said “well, you know, pulling out of this place seems to result in a lot of suffering, so let’s not do that”.
It’s also definitely not clear to me that dogmatism and fanaticism won’t be long-lasting by default (as opposed to, as you say, only lasting for a few hundred to a few thousand years). I expect AI-powered value lock-in (as Wei Dai brings up) to be the default, rather than the exception, particularly for religious fundamentalists who are commonly, actively encouraged to never put themselves in a position to question their faith.
I think the most obvious best case scenario in the case AI2040 gives, with an obviously aligned model, is something like handoff to the ASI model. But, I understand this is less politically feasible (it’s not obvious to me what proportion of the universe you would need to promise to current people to make it politically feasible, but I suppose I’d be surprised if it were 100%).
I really liked it; I think it was very well-written. I have a quick response to the epilogue: https://www.lesswrong.com/posts/EANs7YerYXmaXF9FE/don-t-normalize-a-permanent-underclass-even-a-rich-one-1. But, overall, thank you so much to the AI Futures Project for all of the hard work that went into this!
I’m excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We’ll try to monitor it and respond to many of the comments.
It would be nice to link a supplement with some actions that people with different levels of resources can take to make progress on this. I’m sure this is coming, hopefully soon for momentum purposes.
We have a get involved page for verification in particular. It tracks the current state of play wrt verification and what still needs to be done: https://ai-2040.com/supplements/verification-plan/get-involved
We might do a blog post on other actions as well.
I wonder how the AIFutures’ team’s timelines have changed (esp after Fable, Sol etc)
I’m at 2029 or 2028 median now, not sure. We’ll try to do a reassessment of timelines soon and get out an update.
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
Thanks for the comments!
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there’s a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we’ve burned our entire lead, right?)
This isn’t true because by 2040 in the scenario we are very confident there’s no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we’re less sure). Or in other words, it’s only true if you assume that we can’t detect covert projects after they are started, only right at the beginning when they are diverting their chips.
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
Hi! I took some notes as I read. I’m sure some of them are addressed in what I didn’t read (I only read Plan A, no supplements), or I didn’t pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully or accidentally leak it to other parties?
Could the US government just decide not to give a dividend? Is there some other use it would rather put the money to? Could it, if it wanted to? Would the public trust blind-verified sousveillance?
If the dangers of AI persuasion are countered by having your own AI, this seems to only move the battlefield up one layer—now AIs are trying to persuade other AIs. Do we know if the asymmetries favor defense here?
Is a basin of sanity a coherent or evidenced concept? I know that in the rationalist sphere we like to think we’ve gotten the path to sanity down, but, is it guaranteed that we’re right about that, or that it is memetically dominant, or that the AIs people use will guide them to it? To put it differently—would, for instance, a far leftist or rightist agree with you on what “sanity” means?
Does humanity actually know how to keep a promise and uphold a contract over the lightcone, or at least long enough for aligned ASI to enshrine it?
Incentivizing safety research: outer loop alignment failure? (Sorry, I can’t remember what this one means. Outer loop refers to the human decision making process, I’m pretty sure.)
Handoff assumes that we really truly have solved alignment once and for all. But, are we sure there are no Old Ones lurking in the unknown unknowns, no black swans to ambush us, etc., with the full might of a decade of humanity and vaguely trustworthy AI bent to the task? (I imagine the answer is simply “that’s the best we can do.”)
Insider: Wasn’t the kludge result from last December? I notice more broadly that a few concepts that are established in current safety discourse get brought up several years later as seemingly-new discoveries. I get that you can’t model discoveries we haven’t made, but swapping in ones we know of already feels a tiny bit off.
Insider: It’s depicted that the AI can be taught defensive-only cybersecurity with enough data filtering. But is it actually theoretically possible to have something that can do cyberdefense without cyberoffense? Or at least so differentially so, that defense is dominant in all attack theories?
How do you stop an AI from reasoning further than a few weeks?
Insider: is “one deduction from the frontier” actually safe?
“Adversarially robust neuralese”—does this make sense? Can it be assumed to be real?
The handoff is quite fast and centralized. If everything is so nice and tidy in 2038, are we still in such a precarious state that the deal could still break at any time, and a fast handoff is warranted?
Some further notes from after reading:
Is a “Long Reflection” best described (and dismissed) as a “Long Memetic War”?
What’s going on with mass surveillance in this scenario?
The scenario seems to assume verifiability is the only way forward for capabilities.
What’s been happening with cybersecurity by the time of the pause?
Generally, can this scenario be described as aligned to humanity? It seems like a lot gets gradually handed off to the AIs that I’m not sure I’d be fine with handing off on reflection. Did disempowerment already happen in 2031?
The scenario sort of feels like a bargain with the accelerationists.
What is your way of addressing algorithmic improvement? Is it robust against all scenarios?
Feels funky to suggest this, but have you considered using frontier AI to analyze and red-team your model?
Could a finer grained model of robot development help here?
The scenario seems to assume that no single critical mistake on our part, that the AI can sneak past us, is enough to end the game in the AI’s favor.
I chatted with Fable after I read the piece and I remember adding expansions to a few notes, but I tried to remove ideas it came up with when writing this up.
A key part of this strategy is deterrence, “Mutually Assured Compute Destruction” which gets its own section. It doesn’t mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1⁄3 of the MAIM discussion on verification and how to do this in a multilateral way.
Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It’s a bewildering unforced error—it’s extremely related, it’s a certainly a nontrivial idea, and I told you all this recently—I hope you all fix it.
Sorry that was an oversight, we’ll edit to include a footnote citing MAIM.
This is really thought provoking work. I think the power of personal diplomacy may have been overplayed at the point where ‘the President’ and Xi are mentioned hashing out capabilities limitations over phone call. I understand it’s an optimistic scenario, but these things are generally procedural. Is that a baked in assumption?-- that the standard means of accomplishing international cooperation will have to fall by the wayside in favor of a more gung-ho, personal style of diplomacy if something like The Coalition is to be achieved? I’m generally interested in how you guys made your assumptions re: where incentive structures could carry the day vs. where more concerted political will would have to be built.
Thanks for this important work. One thing I remain uncertain about is whether the scenario’s rapid physical and economic scaling rests on multiple unstated (mutually dependent) assumptions:
The AI infrastructure boom must avoid a major investment correction before AI systems generate enough real economic value to justify the expenditure. If returns arrive later than expected, overcapacity, defaults, or a broader financial shock could sharply reduce capex and produce something resembling an AI funding winter.
Compute expansion must not be bottlenecked for years by grid connections, power generation, HBM, advanced packaging, fab capacity, or fab equipment. You model some of these constraints, but I am not convinced that the physical buildout can proceed on the implied timescale.
Robotics must become capable of reliably constructing and operating power plants, fabs, mines, and robot factories without depending on unpredictable breakthroughs in embodied autonomy: long-horizon reliability, dexterity, continual adaptation, and generalization to novel real-world situations. This seems partly bootstrapping-dependent, because rapid robotic infrastructure growth is then used to justify the compute growth enabling further AI progress.
How sensitive is the scenario to these assumptions? In particular, what happens if there is a multi-year capex contraction, an energy or semiconductor plateau, or a substantial lag between cognitive automation and broadly capable robotics?
These things point me to worrying more about the economy than these loss of control scenarios. Maybe you can correct me on that.
It’s possible that there’ll be an AI funding winter in the next few years but we don’t think there will be. The AI companies really do seem to be close to automating AI R&D and more generally close to automating huge swathes of white-collar work; insofar as this is true the valuations and revenue of the AI companies will keep growing very fast instead of plateauing.
“Must not be bottlenecked” that’s a strange way of putting it—it IS bottlenecked by those things, but at least based on our understanding the bottlenecks don’t bite hard enough to prevent the growth rates we are talking about.
The robots thing is correlated with the AI R&D automation thing. If you buy that the AI companies are close to fully automating AI R&D, then you should also think they are probably close to having robots that can reliably construct and operate power plants, fabs, mines, etc.
@Daniel Kokotajlo, @Vladimir_Nesov I think that 1) would benefit from an extra exploration.
At the time of writing, EpochAI predicts that Hyperscaler Capex will Exceed Cash Flow by Q3 2026. The total hyperscaler capex in Q1 2026 was $148.4B, while Anthropic’s annualized revenue in May was $47 billion, letting one estimate the revenue as $12.5B per Q2. I suspect that Anthropic’s part of hyperscalers’ spending is more in Q1 than the gain in Q2, preventing the capex’ further growth beyond the 23%/yr (and having efficiency fail to grow more than 1.26 times a year, causing the total result to multiply by ~1.6 times a year instead of ~2 times a year. The METR scaling laws relied on the old trend of compute scaling)
What metrics could one use to understand when AI R&D or huge swathes of white-collar work do end up automated? For example, if automating legal work requires a model with an ECI of at least 170 or the 80% METR horizon of at least 10 hrs, then one could rule out that happening in Q3 2026.
Additionally, we would have to rule out the collapse of the USA’s economy and Taiwanese chip industry caused by exogenous shocks like the Iran War causing a blockage of the Bab-el-Mandeb strait and the Taiwan War.
The main point I’d disagree with here is that AI R&D ⇒ AI Robotics ⇒ Robotics-driven economy. I’ve spent a lot of time in industrial environments and the problems are never as simple as people think they are—and cannot be handwaved away like a software feature might be.
To be clear: I do think that AI Robotics is the future, but the aggressive timeline predictions that I’m seeing like AI 2030 etc. seem overly bullish because real-world adoption will almost certainly be slower than people in the research labs think. Even in the case where you somehow manage to get a large enough flywheel going that you essentially cut out everyone within the factor and have factories autonomously building themselves (very unlikely—the scope of problems within a factory is incredibly broad such that even generalized human intelligence is often not enough; you need domain experts for many basic tasks. The ‘basic’ task of sheet metal folding, for example, is brutally difficult as an ML problem.) it would take years in the extremely aggressive case to get these factories up and running, with permitting and raw resource/logistics timelines serving as major bottlenecks. And this is before we talk about the incumbent competitors for current demand.
I often see that researchers have a tendency to handwave away these ‘more basic’ problems within their research—which is all well and good, because research is about asking questions, and you’re best equipped to do that when you can remove a bunch of other confounding factors—but when you start talking about real world hyperscaling and real ‘rubber meets the road’ problems, the stack of your necessary dependencies grows exponentially and you cannot skip over steps like one might avoid a out-of-scope software feature or leaving a proof to the reader.
Granted, I’m not saying that robotics cannot succeed—but simply that the timeline is longer than I think people are expecting. Look at the timelines on industrial companies versus SaaS from the same era: SpaceX was founded in 2002, before Reddit, LinkedIn, Facebook, and is only just now reaching IPO, whereas the SaaS companies were household names nearly a decade earlier. Uber was founded in 2009 and took until 2023 to become profitable. In general, industrial timelines are slower because the real world works at a slower pace than software. If I founded the next big industrial robotics company today I would expect it to take till 2040 until it reached maturity—and we’re not yet at the point where I would say robotics has hit the inflection point for that.
The shutdown section is missing a major possibility: AI shutdown combined with hardware bans. Deep ultra-violet (DUV) and extreme-ultraviolet (EUV) lithography require highly specialized machines that there are a limited number of. So along with the shutoff of AI research, also shutoff production of new DUV and EUV lithography machines, and then shutdown the remaining machines. This would probably take years, or require significant payments to motivate companies to do so, but at that point the existing stock of high performance GPUs would start to decrease thru attrition, and the remaining ones would start to increase in cost since no new ones are being produced. Buy back and destroy programs, or programs to buy back the existing GPUs and shift their use to non-AI use such as scientific simulation would increase the speed this happens. This would basically force new technology back to circa 2005 levels, which would be much less risky for creation of super-intelligence. (Or back to a different level, depending on where people decide to do a hardware ban. And it is worth mentioning that computer use has changed significantly because of increases in the amount of data that can be stored on magnetic media and in the amount of fiber optics that have been installed and this would not be affected by a lithography process limit.)
This would be politically much harder than a pure AI shutdown (since it affects all computers), but would solve the problem in Plan S of “The main downside of Plan S is that the shutdown deal will probably break down eventually.” since decreasing the amount of compute available can decrease the chance of creating artificial super intelligence.
(As a related example, https://ifanyonebuildsit.com/treaty restricts 28 nanometer process node and smaller technology)
The effectiveness of this strongly depends on how easy it is to rapidly rebuild the semiconductor supply chain; so I’m interested of an analysis of this question. I think that this is a promising potential proposal.
I haven’t seen an in depth analysis, but I also have not looked hard. I did read today: “One pivotal question is how far China has progressed in building its own EUV machine. Reuters, a news agency, reported in December that a team of former ASML engineers in China had completed a prototype EUV machine in 2025 and were testing it at a high-security lab in Shenzhen. ASML said it cannot control where former employees work, but they are bound by confidentiality agreements and in some cases the company has successfully taken legal action in response to theft of trade secrets.
The prototype had not yet created any working chips but the Chinese government had set a goal for it to produce functional ones by 2028, Reuters reported. ”
https://www.economist.com/china/2026/07/05/has-china-obtained-the-worlds-most-important-machine
So basically China has been trying to build a EUV machine since roughly 2019, and still has several years to go. DUV would probably be relatively easier but still fairly difficult since visible light optics and mirrors cannot be used. Current semiconductor manufacturing requires thousands of steps, all of which have to sufficient yield for the final output to work.
The China Integrated Circuit Industry Investment Fund spent around $20 billion dollars in Phase I (2014 − 2019) to work on building DUV technology: https://en.wikipedia.org/wiki/China_Integrated_Circuit_Industry_Investment_Fund
Visible light photolithography semiconductor making can be done in a home shop as has been done by Sam Zeloof for example: https://www.hackster.io/news/etching-pcbs-sam-zeloof-steps-it-up-a-notch-and-down-a-few-orders-of-magnitude-with-his-own-ics-cdb0e297bc1a
So EUV level even in the easy case where you have access to employees who have worked on the machines and access to parts of the existing supply chain and likely state level support is still nearly a decade long project.
Overall I like this plan and think it raises good points and shows possible paths. Thanks for the great work.
1) The pause time shown is from 2035 to 2040, which feels OK. One concern I have with longer pauses is the usual innovation of HW and SW, which tend generally to range from 100 to 1000X per decade each. So a 5 year pause means all non-controlled AI get 100-1000X more compute per dollar—systems that cost $1 billion to train and run well drop to 10 or even 1 million, which adds to the problem of enforcing a ban. You may want to note this cost reduction in the 2037 or 2038 discussion of Plan A.
2) I agree that working to build an aligned AGI that is aligned well enough to survive recursive self improvement through to ASI is key and we should be starting work on it now. I would recommend citing Hinton’s comments about maternal instincts and Contemplative Wisdom for Superalignment as possible examples.
3) I think it’s worth mentioning the potential that AI’s either achieve consciousness or it becomes clear that we can choose this through design. If AI’s ‘naturally’ achieve subjective experience as they pass through true AGI, then their welfare becomes important. If it turns out that with selective training or design we can ‘choose’ whether AGI is conscious, we should talk about and plan for how we collectively want to handle that. Of course this topic could become a huge discussion post of its own, and I don’t think you need to detail out all the ways this plays out, but I think it should be mentioned with a paragraph.
I don’t think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.
Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viability
Objection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.
Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.
Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.
I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.
I’m very impressed by the proposed Total Research Transparency. I actually found it appealing even beyond the many reasons mentioned in the plan. It takes advantage of key properties of the current training paradigm, and this is actually desirable, because these properties are likely to remain in future AI systems:
Model training and serving will keep incentivizing a small number of large neural networks, with agent diversity coming from context. Efficiency of batched inference is too large, the learning algorithm and the hardware have co-evolved around it. My opinion is that continual learning will be hard on hardware like the current one because of loss in batching efficiency. And radically different hardware will take longer to develop.
The auto-regressive architecture will also likely remain, because having a chain of tokens empowers the model with the ability to perform inherently serial computation, making it strictly more computationally powerful. This makes it possible to create verification schemes with teeth (via CoT monitoring etc).
So open sourcing the algorithms, closing the weights and monitoring inference seems like the right balance.
If we create an actual recommendation from this idea, how can we actually get labs to cooperate on it? AI companies will likely dislike it, but for example Boaz Barak actually endorsed the push for increased transparency in this tweet. I hope the authors have a plan to push for it beyond this writeup.
For the past months, I’ve had many sleepless nights thinking over a scenario which I couldn’t resolve completely, a scenario that involves a “sovereign leap” by smaller countries in their last moments to combat permanent disempowerment. They deduce that the best means of balancing the asymmetry of superintelligence is creating weapons of biosphere destruction.
They precommit to destroying the biosphere if the US and China try to permanently lock in the status quo in their ASI deal, or destroy or poke their biological weapons in any way. There would be mostly two paths that the US and China together could take.
Path A:
The US and China call this a bluff but do not act hastily first; they direct their army of ASIs to subtly overthrow those countries to ensure they would not be able to poke their noses in their terra kingdoms. Over the course of the year, several countries are disempowered, overthrown, and taken control of. The moment of glory for the battle won is sound, and everyone in the US and China is happy knowing that they will not die and will live forever. But there seem to be subtle anomalies involved in forest areas where no one would look before, for there were no people living. Some form of matter seems to consume hectares of forests and leave some form of persistent plant-like shapes. The bilateral ASI screams danger and, in a concerned effort, tries to save humanity from extinction. With the help of nanotech, it starts to terraform the solar system. Mars and Titan are the slated homes for a new humanity. As a backup plan, it also creates embryos and hibernates several IQ-boosted humans with nanotechnology to terraform the planets and live on. So, it sends them off to different parts of the universe in hopes that one of those seeds survives.
Path B:
The US and China take it seriously. They direct their army of ASIs to debate the best way they could be transparent with those countries to ensure that they can trust them. After weeks of superintelligence effort for solving a decision theory, the ASI comes up with a plan that the US and China leaders can’t know in advance, for they would be able to sabotage it for their own benefit with 90-95% success, but even a 5-10% probability of extinction is unacceptable for the ASI, even though it might be acceptable for the leaders of those countries. So, it invites all the leaders of those countries along with their ASI supervisors. It sets up a supervision and country alignment plan to create ASIs that are aligned with CEV cosmopolitan humanity with no bias in either direction. What would that look like? I will answer with the lazy “you wouldn’t predict a Magnus Carlsen move” type of answer. Perhaps because I am not smart enough to come up with such an alignment plan.
Is this unreasonable? Do you think that, in reality, state-level actors could not develop such weapons with the help of proto-AGIs before the Plan A deal?
I just noticed this part of plan S.
This seems like a mistake. Whether plan S is good or bad will depend on if the conditions for AI development are better or worse when it finally ends. If fabs are allowed to scale up and research on how to make cheaper and more efficient chips is allowed to continue, then we’ll plausibly have much more compute and a much faster takeoff when plan S eventually ends, plausibly making it all net negative. Chip R&D in particular might actually be the most important thing here—Moore’s law continuing for several more decades would make takeoff much, much faster if it the deal eventually broke down.
Depending on how much political will there is (and in plan S there is presumably a lot!) I’d guess it’s better to go the other way and reduce chips and fab capacity, and to ban R&D into producing more efficient chips. So that if someone ever leaves the deal, it takes maximally long for the world to resume the singularity. (Leaving enough time to coordinate on another deal, or to do research, etc.)
In fact, if you expect non-tracked compute to decrease over time (e.g. culture change makes it more and more outrageous to hide chips; replacement of leadership and rank-and-file at various places give greater probability that someone changes their mind or leaks; it’s tempting to run the chips in ways that would degrade them or to sell them to buy-back programs; more total intelligence agency effort accumulating over time) then plausibly an “out” of plan S would be to restart the singularity a while later but with much less compute in the world, capping the speed of the singularity in a much more robust way than plan A.
In order to reach the other more aggressive endings (Plans B/C/D), does humanity have enough electricity to currently fund more expensive/larger training runs?
I think there’s a problem underneath this that isn’t really about capability or timelines. Even today, entry level jobs are shrinking and students are collecting credentials and work experience from first year, before they’ve really learned anything, just to stay competitive. Nothing wrong with that individually, but it’s a race, and the thing it’s racing against is a machine that can eventually do the job better anyway.
The economy is already restructuring around this, not in some future scenario, now. Figuring out what a computer can do increasingly means figuring out what a human’s role actually is, not as an afterthought once alignment is solved, but as its own open question. Even in a good, safe, slow scenario, we still haven’t answered what a person’s role is once credentials and experience stop being the advantage they used to be.
If you had to hash out a super-aligned AI scenario with respect to AGI-enabled dictatorship vs. abundance for humanity, what might that look like?
And why is it (obviously/reliably) Anthropic that will capture mass compute share in Plans D and C (i.e. ASI Race scenarios)? Or just a placeholder?
The two-layer design is right, and there is a physical reason. Ideas are information, and information leaks: the US tried to export-control cryptography in the 1990s and failed completely. Compute is physical, and physical things can be verified, the way fissile material was under the NPT. So open research plus restricted hardware is the right split. But it means the whole plan depends on one variable: does frontier AI stay compute-hungry? If algorithmic efficiency keeps letting last year’s frontier run on ordinary hardware, the world is licensed on paper but leaky in practice. That variable should be the plan’s primary tripwire, not a background assumption.
My main worry is that the plan defends against the wrong failure mode. MACD deters overt defection: leaving the treaty, building a secret datacenter. But history says the overt move almost never comes. Preventive war against the Soviet nuclear program was seriously advocated and never executed. What actually happens is covert friction: quiet, deniable interference kept below the retaliation threshold. So the real question is whether the verification layer can detect steady covert cheating, and the plan itself admits the gap. Footnote 11 of the transparency plan concedes a defector could do research steganographically through the monitored channels, and the misuse defense depends on refusal training, which jailbreak research shows is reliably bypassable.
The pace mismatch makes it worse. Nuclear arms control took about twenty years to reach real verification, for a technology that barely changed while being negotiated. AI capabilities turn over in months, so a treaty will always describe last year’s AI. That’s why AI-assisted verification, their own “AI Silver Bullet”, is not a side project: it is the only mechanism that can re-match the two speeds.
One concrete ask: make the security assumptions as measurable as the compute assumptions. The plan quantifies its compute verification (“99.99% confidence that 99.9% of the compute was verified”), but there are no comparable numbers for refusal robustness or for the steganographic capacity of the monitored channels. Quantified bypass rates and capacity bounds would make the security claims treaty-grade, the way the compute accounting already is. Bounding steganographic capacity in particular looks tractable, and nobody is doing it.
Sorry, one last comment/question:
I’m really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn’t seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn’t show up?
How surprised will you be if this specific technology doesn’t pan out?
The title of the section is “The Apocalyptic Arrival of Truth on Earth.” The lie detectors are there as an example of a social technology that could be invented by AIs that would have big consequences for society. The point of the 2037 section is to try to briefly touch on how there won’t just be more material abundance (more factories, more housing, cheaper cars, etc.) but also new social technologies, new ideologies, new political fault lines, etc. None of this stuff is particularly load-bearing for whether Plan A is a good idea or not; we included it because we were writing a concrete scenario and it seemed like this sort of societal transformation would probably be a consequence of all that AI-powered research etc., and so it seemed like we should write about it.
I currently think that lie detectors of the sort described are maybe like 2/3rds likely in the sort of scenario described (i.e. pause at top human expert AI) and closer to like 95% likely with superintelligence.
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
In the future we are describing, AI is big enough that new power generation will need to be constructed to handle it, and so we might as well build that new power generation in Mongolia etc. too alongside the datacenters.
I have read the entire post and sub-scenarios and know I am not able to gauge the reality and speed of what is said to lie before us. Nonetheless, I am in shock.
Another Tolkien quote “”It does not do to leave a live dragon out of your calculations, if you live near him.”
And so we do.
Questions/thoughts:
What will be the cost of a bag of potato chips if everyone is making $10MM?
What the heck is all the work being done?
What is the cost to the environment with such large GDP increases?
Will anyone have children?
What happens to all the parchment books squirreled away in dusty libraries? Do they get scanned and their information added to the training data?
Simulating past histories seems not far from Nick Bostrom’s simulation trilemma scenario.
The notion of radical transparency and truth fascinates me. Imagine:
MoTru. The Museum of Truth implemented by neutral curAItors, AIrchitects and desAIners.
All Sides or Less Wrong becomes the AI-driven trusted global news network.
Public and private sides of life, politics, and culture dissolve. Everyone knows everything about everything, or could, and an the Machines of Loving Grace era is a result.
I lean toward think a bad outcome will with bad actors shaking hands with fingers crossed. They go back to their tunnels and build the killer AIs. I say this as Trojan Horses have been rode throughout the course of human history.
Note #185 is misleadingly worded. It sounds like the probes could reach the entire reachable universe in 6 hours, whereas the cited paper says 6 hours of Dyson-sphere output could power their launch.
I think the induction trap in your entire premise, is assuming an independent actor isn’t going to engineer something without asking anyone’s approval. The open source community is a vast repository of functional prototypes. While some people sit around debating what should happen, other people are investing time into actually building the future.
Removed. Wrong Think. Good luck!
Why do you think this is helpful? The way you’re deploying it, it seems like a fully general argument against any form of foresight. Obviously the essay is pretty good and a useful lens but using it as a fully general dictum/certainty seems ill-advised.
Also von Neumann is a smart dude, but not, like, a prophet or anything.
Removed. Wrong Think. Good luck!
Imagine someone reposting random vaguely relevant Bible verses to essays you read. And then when you complain, they respond with an AI explanation for why that vaguely related Bible verse is relevant. You’d probably find that annoying too!