I’m broadly interested in AI strategy and want to figure out the most effective interventions to get good AI outcomes.
Thomas Larsen
Karma: 4,474
How to pace the US frontier
Plan A depends on the US government deciding to change the status quo, so probably won’t happen regardless of the timeline. I’m in favor of pitching the Trump administration on it as well as any future administration.
I think the viability is higher later because there are larger effects of AI on the world and therefore more political will for large and unprecedented actions. But of course that might be too late, the timeline might be very short. Also, even if the timeline is >2030, getting started early is super important because covert projects are much more worrying if we start Plan A deep into the intelligence explosion (e..g at AC or afterwards). I think if we start at 1 year before AC, like we do in the scenario, things look very good here, but if we start significantly before that, things look even better!
So basically I think doing all the paths in parallel, with the goal of getting something like Plan A done as soon as possible, is good. (so long as there aren’t big negative externalities of doing so, which I am also pretty worried about)
IMO this variant of Plan A is one of the ~5 plans that are competitive with the version of plan A in the scenario.
Thanks for the comment!
1. I think the AIs might be philosophically competent enough to solve ~all the problems, and using the AIs to solve them is basically the right move. We try to make this clear at the start of the epilogue. We wanted to be somewhat more concrete in the scenario than meta level solutions like this, and try to sketch out in concrete detail what the solutions might actually look like, which is why we didn’t just stop at “handoff to the AIs”, though I do think in practice we should mostly be handing off to the AIs at this point.
2. I agree that making the AIs this philosophically competent may be very difficult and not happen by default. I think this is indeed a big concern, and I wish we’d written (and thought) about this more carefully. The place we’ve written the most about this is here: https://ai-2040.com/supplements/alignment-roadmap#phase-4-handoff (which TBC is written from a Plan C perspective, assuming much less lead time than Plan A, and is the supplement from which we linked your post). In Plan A, the story is largely that we have a huge amount of time with ~human level AIs, and many people have access for many years, and those people will (hopefully) make progress on this.
The effectiveness of this strongly depends on how easy it is to rapidly rebuild the semiconductor supply chain; so I’m interested of an analysis of this question. I think that this is a promising potential proposal.
Thanks for this pushback. I think you are right about the top level that generally it was too angry, and decided to edit it to tone it down (but preserved the original and the diff here: https://docs.google.com/document/d/1i6Y9KTK2cyJ7y3XQtGY6f9F4nloRQr-rep8qSsicYUA/edit?tab=t.0 for transparency). I’m sorry about that.
Re your specific example, I still think that list is pretty misleading about what I actually think, so I do still want to set the record straight.
It does seem more valuable to litigate the economics disagreements on the object level, but we need clarity on the views themselves to do that productively. I think the econ supplement to ai 2040 is the best place we’ve done this so far: https://ai-2040.com/supplements/economics-of-plan-a
After reflection, I decided to remove the two false representations that were the most ambiguous from the top level post. They were:
>A cadre of elites decides which research directions are permissible, caps global compute and robotics, and creates state-administered scarcity rents.>
It feels like the same plans as I’ve been hearing about for nearly a decade in the AI safety community, but filled with more details. In 2018 when I worked for the UK government, a prominent AI safety research organisation told me that “we need to solve the technical alignment problem, and then simply hand it to the UN to implement everywhere.” This was before the field invested in governance and politics; Plan A broadly similar, but with all sorts of mechanisms to fill in the gaps.I think these are still wrong/misleading, but are less clear cut than the examples still in the post.
(We also edited the post to be generally be less combative, because I think it was too combative and regret that; see the italics at the top; you can see the diff here)
(a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
As I said in the post, this part is largely a strawman of our views. The part I mostly meant was “Nor do I think we will get real GDP growth of 50% in 2032” and “the model moves far too quickly from AIs being able to perform tasks to robots being reliable, legally deployable, organisationally integrated substitutes for almost all labour, and from there to a closed-loop reproduction of capital”. Perhaps what he means is that there will be delays, but eventually we’ll get 50% GDP growth?
I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.
I tend to find the exact opposite is true. People love claiming that the real disagreement is real world bottlenecks, but usually its really differences in capability expectations. I think the AIs will be wildly superintelligent, with vast quantities operating at the equivalent of 100x or 1000x human speeds, also with a much higher qualitative intelligence due to things like knowing way more than any human can know, having much more experience than any human can every get, and having a physically much larger and more connected brain such that they are able to make discoveries and model things that are far too complicated for humans.
This is obviously an extreme milestone; we outline earlier milestones here: https://ai-rates-calculator.vercel.app/, for example. I would encourage people to make forecasts of when they think these specific milestones will be crossed (which might be “in hundreds of years or never”). If earlier than that, maybe its just a timelines / takeoff speed disagreement.
I kind of expect that many people would argue that this is the wrong ontology for thinking about AI capability progressions, and that actually, it’s somehow going to be more diffuse/multipolar or something, in a way where thinking in these terms isn’t useful for modeling the world, in which case I’d love to hear a better frame.
Still, I think that most people with very different views would tend to disagree that the notion of ” superintelligent” AI that I outlined above would happen but will have small effects on the world. I think this view is extremely hard to make coherent. Once we’ve got AIs like that, the cognitive labour supply would become enourmous, such that almost all the cognitive labour happening would be AIs, not humans.
I am a co-author on AI 2027 and agree with the main point here. See this shortform where I say something similar: https://www.lesswrong.com/posts/q8fdFZSdpruAYkhZi/thomas-larsen-s-shortform?commentId=ayQsdj35GfCb6XLKB
“I think that the AI behaviour after the AIs are superhuman is a little wonky and, in particular, undersells how crazy wildly superhuman AI will be”
I argued for this at the time and continue to think it was a mistake to not have the ending be much crazier and involve superpersuasion/nanotech/etc to a much greater extent.
I think it’s true that Séb doesn’t believe in AGI/ASI in the sense that I mean: see his fourth response here. Also see e.g. this model he posted which argues that comparative advantage implies postASI humans will still have jobs (which doesn’t make sense if you think ASI can use the inputs that humans require much more efficiently than the humans do).
Seems very reasonable to be skeptical of many of AI 2040s proposals despite believing in AGI/ASI. I’d be curious to hear more (though perhaps you should post those on the main post, not here, if you feel interested :) ).
Re clarifing the quotes; reading them now I still think they are all misleading but to varying degrees and probably won’t spend more time further getting into it because there’s a lot on my plate right now, but if others think it’d be useful for me or someone else at AIFP to do another pass here trying to further clarify let us know (e.g. by reacting or commenting here).
You might be interested in other proposals we make such as filtered transparency (see here). I don’t fully understand your comment but I do think that TRT is a bigger lift than filtered transparency. We recommended it because I think it is the best option, and I think it’s viable if enough people push for it. But TRT is NOT a load bearing part of Plan A; more lax transparency proposals are very much consistent with it.
Thanks for this comment!
I think the central disagreement is about whether AGI/ASI is real or not, and almost everything else is just downstream of that. The claims and policies made in AI 2040 only make sense in a world where you can have AIs that are much much smarter than humans, and that mere AGIs can cause explosive growth.
Given that Seb disagrees with the economics substantially, I would guess that he thinks that AIs will never reach the point where they can fully automate the robot and semiconductor supply chain; if we do get full automation then fast exponential growth seems to quickly fall out of reasonable modeling. Unfortunately I don’t understand his views that well, so it’s hard for me to say exactly where he thinks AI capabilities will cap out.
Re: Transparency and government powers, I think his post pretty badly misrepresented what we were saying, where I think someone who read his piece but not ours would be extremely misled about what sorts of regulations we were proposing. But I also imagine there’s substantial policy disagreement there… so from my perspective it seems like both?
I will say that with government powers in particular, Plan A does involve governments into AI more than the status quo. Insofar as you think that’ll be predictably bad, that is a genuine downside of Plan A. There’s a sense in which any regulation is “central planning”… and yes, we do advocate for certain types of AI regulation. But overall in Plan A we aim for as market based and decentralized solutions: for example, we propose several cap and trade regimes, which seem to me the maximally libertarian way to go about this regulation, and so calling it “central planning” seems pretty misleading. Other regulation is more difficult to set up this way, such as limitations on algorithmic progress.
Thanks! My intuition is the opposite, but I’d be curious to know which interp techniques you think are most valuable right now.
Though actually I realized that paragraph in the post is somewhat misleading (in a way which strengthens your point), sorry about that! Total research transparency gives you ~all the code, and some of the training data, but most of the training data is in the opaque database, and so can’t be fully audited except via AIs, which might still get you a bunch of the benefit. (Read our detailed proposal here: https://ai-2040.com/supplements/transparency-plan. )
Our response to Séb Krier on Plan A
[AI 2040] Transparency Plan
Thanks for the comments!
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there’s a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we’ve burned our entire lead, right?)
This isn’t true because by 2040 in the scenario we are very confident there’s no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we’re less sure). Or in other words, it’s only true if you assume that we can’t detect covert projects after they are started, only right at the beginning when they are diverting their chips.
I agree with the overall point! We tried to think through a bunch of technological progress but no doubt were missing some important stuff.
See, for example:
- Military power
- Robots (mentioned throughout the secnario)
- Lie detectors
I haven’t looked into BCIs much at all it’s plausible they should play a bigger role earlier.
I think this consideration goes the other way, unless I’m misunderstanding. Algorithms diffuse to covert projects, but compute doesn’t, in the Plan A scenario, we deliberately scale compute for this reason while heavily limiting algorithmic progress.
(Also note that the 1.5M H100e is before accounting for detection)
People often analogize international deals on AI to traditional arms control agreements (e.g. New START, London Naval Treaty, etc). This analogy holds up for some potential proposals (e.g. directly doing arms control on GPUs), because they are close to one off agreements with ongoing inspection / verification.
For Plan A as described in AI 2040, I think a much better analogy is to the series of allied planning conferences during WW2. Some important similarities:
In both WW2 and Plan A, the situation is complicated enough that it doesn’t make sense to try to plan for every eventuality at the beginning and commit to a one-size-fits all plan, it needs to be iterative. The allies had to ongoingly coordinate complicated logistics, military planning, etc. In Plan A, countries will have to ongoingly make agreements like: exactly how much algorithmic progress to allow, which training runs should be allowed, if there is evidence of defection, how to punish that and/or improve verification, if there are AI incidents, how to respond to them, how many robots and GPUs should be allowed to be built?
The world is transformed. In WW2, e.g. the car factories are retrofitted to build planes, in Plan A, the car factories are retrofitted to build robots. There was overall rapid technological change during WW2; there will be even faster changes during Plan A. In Plan A, a typical human experience is: working a normal job in ~2030, working in very highly paid jobs in robot factories in ~2033, and being technologically unemployed by 2035.
The deals at the end of both need to resolve issues that are hard for people to even imagine at the beginning (e.g. the division of space, loosely analogous to the creation of East and West Germany).
Note that the allies did not trust each other in WW2 (especially the US and USSR).