Our response to Séb Krier on Plan A
This criticism of AI 2040: Plan A by Séb Krier unfortunately seriously mischaracterizes our proposal. It also mostly contains flat assertions, not real argumentation, and the argumentation in it seems quite weak. While we appreciate constructive criticisms of Plan A, such as the ones by Tom Davidson, Richard Ngo, and 1a3orn, we feel the need to correct the issues in Séb’s response. First, we’ll go over the specific false representations, and then we’ll give a point-by-point response.
False Representations
I’m not claiming you shouldn’t prepare and improvise in the dark, but rather that this version of preparing bakes in too much and leaves little space for the effective but uncomfortable trial-and-effort that real life requires.
The exact opposite is true. Plan A is extremely iterative. In the status quo, there is trial and error, but ultimately companies aren’t going to choose the safer or more societally beneficial path, they are going to choose what the market wants. In Plan A there is much more time for AI companies to gain evidence and for governments to respond reasonably to the sweeping changes. Thanks to total transparency and broad deployment, all of this evidence is accessible to academics, independent researchers, and the public instead of being sealed away in the labs where only lab insiders can see it. Our plan maximizes learning and room to experiment.
I think it gets a lot of economics wrong. It reifies a number of views common amongst Bay Area technologists, which include (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
Much of this is wrong. Specifically, we believe: (b) not all profits will accrue to labs,[1] (c) that Plan A would significantly decrease lab profits,[2] and (e) ‘de facto’ nationalization is not an optimal response.[3]
The authors themselves note that “there isn’t a central planner with authority to regulate AI worldwide” as a fundamental issue with the status quo.
This is an out of context quotation to make us sound bad. You can see this quotation in context here. We note the absence of a global central planner as a constraint Plan A must work within. The Deal we propose does not create such a central planner, nor do we think this is a fundamental issue with the status quo.
A cadre of elites decides which research directions are permissible, caps global compute and robotics, and creates state-administered scarcity rents.
In Plan A, American and Chinese regulators do coordinate to ban egregiously dangerous research directions. But because of total research transparency (TRT), all of the evidence that informs the decision to ban a new training method is public. Anyone can inspect the evidence and check the regulators’ work.
It feels like the same plans as I’ve been hearing about for nearly a decade in the AI safety community, but filled with more details. In 2018 when I worked for the UK government, a prominent AI safety research organisation told me that “we need to solve the technical alignment problem, and then simply hand it to the UN to implement everywhere.” This was before the field invested in governance and politics; Plan A broadly similar, but with all sorts of mechanisms to fill in the gaps.
The one-sentence policy proposal Séb recalls hearing eight years ago is not at all what we are calling for. In our scenario, private AI companies iteratively solve the alignment problem while regulators set a low floor by banning the most obviously dangerous AI R&D practices. No one alignment plan is forced on all the companies from above.
Apparently it’s ‘surprisingly easy’ for the rest of the world to get buy-in for a technological slowdown because the only three possible outcomes of AGI development are “take our jobs,” “cement hegemony forever” and “get everyone killed.”
See this quotation in context. We do not say, as is implied here, that these three possible outcomes are exhaustive. The whole point of writing Plan A was to increase the chance of a better outcome, namely a peaceful, multipolar world where the benefits of AI are shared widely.
Point-by-point response
Overall the response is light on arguments and heavy on sneering. (We’ll point out examples below.) That said, it’s not all like that. There are some substantive objections and arguments made. So we’ll now go through and respond to everything.
The kind of world a particular AI safety milieu reaches for when asked to imagine a positive future really illustrates the extent of the field’s monoculture problem.
We think this understates the originality of Plan A. Our policy recommendations are certainly not the same thing you’ve been hearing from the AI safety milieu since 2018. For instance, where else have you heard someone seriously propose Total Research Transparency?
But I’m going to take it at face value, and offer nine issues I have with this administrative utopia.
First, I dislike scenario/table-top type fictional writing because (a) they fix some assumptions in time, so you have to basically operate within the framework and pathways offered. Dean made this point previously about an essay on the ‘permanent underclass’. It’s actually not very hard to design a scenario that implicitly favours a particular outcome. And (b) they blur the descriptive, predictive, and normative statements into one big story, and the level of detail can create a false sense of coherence. Richard Ngo makes this case well and Fleetingbits does so too with regards to their previous 2027 piece: a lot of incorrect predictions and theories of government, and the (extremely homogenous) community tend to be very selective of how they assess the track record on the political side of things.
Eisenhower said “Plans are worthless, but planning is everything” and we agree. Plan A as stated will obviously never happen exactly as we laid out. But the same was true of the plans for the Normandy invasion; reality quickly diverged from the plans, and the planners knew that this would happen. But trying to coordinate a massive effort like that without a plan would have been completely hopeless.
It is actually very difficult to design a coherent scenario that favors a particular outcome — over the course of writing Plan A, we often wrote up ideas that we thought were good, but then turned out to be inconsistent with some other aspect of the vision, thus forcing us to choose. Writing up scenarios like this is a huge amount of work, but we would nevertheless strongly recommend AGI companies such as GDM, OAI, and Anthropic invest the effort to actually try to articulate an internally coherent vision.
Re: blurring descriptive and normative, we do wish we could have done better here. We wrote “Plan A Assumptions” in part to clarify these issues, but it’s unfortunately an inherent difficulty of the format. In our view, scenario recommendations are the worst ways to make AI policy recommendations, except for all the other ways, which have much worse problems (such as allowing their authors to avoid making any real claims at all about the future, as is the norm in most AI policy papers).
Second, I’m not a fan of the notion of a big ex ante ‘plan’. There’s a notable difference in the actual strategies that a government might draft, which are typically drafted to allow plenty of optionality and not narrow the option space too much, and plans that are essentially a step-by-step guide, thought through from first principles. The latter might be fine for a very narrow question, but when you’re dealing with something as all-encompassing as the economy and AI, it ends up looking like the worst versions of dirigiste central planning. The authors themselves note that “there isn’t a central planner with authority to regulate AI worldwide” as a fundamental issue with the status quo. A cadre of elites decides which research directions are permissible, caps global compute and robotics, and creates state-administered scarcity rents. I shouldn’t need to explain why this is bad and dangerous, anyone can study History and Economics in their free time. I’m not claiming you shouldn’t prepare and improvise in the dark, but rather that this version of preparing bakes in too much and leaves little space for the effective but uncomfortable trial-and-effort that real life requires.
As we say above, this is a mischaracterization of Plan A. We don’t propose a central planner; our proposal would spread out the power in several ways instead of concentrating it in a single AI, company, or government. There would be more companies at the frontier of AI development, more countries with power over how AI is governed, and more transparency so that the public can see what’s being done with AIs. Far from appointing a “cadre of elites” to control the fate of AI, Plan A enables a wider range of people to be involved in the conversation and have power over how it all goes down compared to the status quo.
As for our plan baking in too much and not leaving room for trial-and-error/learning by doing, it’s exactly the opposite. Plan A buys us ten more years than we otherwise would’ve had to experiment on our AIs, understand how they work, and try out different regulatory approaches before we proceed to superintelligence. And thanks to total transparency and broad deployment, all of this evidence is accessible to academics, independent researchers, and the public instead of being sealed away in the labs where frontier AI employees can see it. Our plan maximizes learning and room to experiment.
Third, much of the essay professes to care about centralization of power. I think the authors (like many in the West, lately) really underestimate the horrors of authoritarianism, and fail to consider how to hedge the risks of institutional misalignment. Commercial concentration is bad, but so far the horrors of the past were more often than not enabled by the machinery of the state rather than through commerce. Building an entire apparatus tasked with maximally empowering the government and its grip on research, knowledge, and technology is dangerous. It may be acceptable if accompanied with real mechanisms to keep state power in check, but unfortunately it hands to the state the very structures needed for deep repression and control. Power is well known to corrupt and somehow we’re supposed to ‘trust the plan’. Liberal democracy is extremely fragile; suffice to say that considering options like ‘network taps’ and shutting down 90% of training compute provides affordances to the state, and the essay provides barely any consideration of how this could be abused. Appendix K tries, but it’s not persuasive, and the verification regime proposed is basically designed to detect defection from the plan, rather than prevent abuse. The fundamental issue is that ‘centralization of power’ is solely understood as ‘labs centralizing power’ - which is also a questionable assumption. I disagree with the characterisation, but even if it were true, the solution to centralisation of power is not to simply switch the actor who gets the reins. See also Richard Ngo’s discussion on a related question.
Again a mischaracterization of our plan, but set that aside because there’s more to say. For one, Séb is underestimating how much power the government already has over AI. He freaks out over us proposing network taps for verification when the government can already see what’s going on in the datacenters if it wants to. Whenever he pleases, the President can hijack a company using the DPA, destroy its revenue source by blocking model deployment with export controls, and so forth. How exactly is Plan A giving them more power than they already have? The point of the network taps is to diffuse power by enforcing transparency — allowing the public and foreign governments to see what the US government can see in the status quo. Everyone gets to see the code that trains the AIs that are eating the whole economy, which is crucial for preventing extreme power concentration. Séb doesn’t engage with this logic in any way.
Fourth, I think it gets a lot of economics wrong. It reifies a number of views common amongst Bay Area technologists, which include (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response. I think each of these assumptions is at best massively overstated. Not completely unreasonable, but way too coarse, and emblematic of a fundamental issue with these kinds of scenarios: they stack highly disputed and out of distribution assumptions (since ‘this time it’s different’), like a highly leveraged bet, and then make highly disruptive and risky prescriptions based on that. Taking issue with the maximalist version of these beliefs does not imply one is not ‘AGI pilled’. Nor do I think we will get real GDP growth of 50% in 2032, particularly if the entire economic benefit that supposedly affords $1M/yr per American person in 2035 comes from state-levied ‘permit revenues’. While I won’t have time to delve into the details of the separate ‘Economics of Plan A’ piece, suffice to say the model moves far too quickly from AIs being able to perform tasks to robots being reliable, legally deployable, organisationally integrated substitutes for almost all labour, and from there to a closed-loop reproduction of capital.
Séb just asserts that we get all the economics wrong without offering any counterarguments or otherwise engaging with our reasoning. Let’s go item-by-item. (a) We’ve given detailed arguments for fast diffusion and economic transformation. (b) This is an exaggeration of our view, though we do think the AI companies will be the single biggest winners in an economy transformed by AI, capturing much more of the profits than, eg, workers or owners of capital other than semiconductors and robots. (c) This is not our view. Our policies — most notably including total research transparency — would dramatically cut AI company profits relative to the Plan D counterfactual.[4] (d) We’ve explained at length why we predict explosive growth soon, and Séb doesn’t engage with our explanations at all. (e) We do not propose nationalizing the AI companies, as Séb would know if he had read our plan.
Fifth, it feels like the same plans as I’ve been hearing about for nearly a decade in the AI safety community, but filled with more details. In 2018 when I worked for the UK government, a prominent AI safety research organisation told me that “we need to solve the technical alignment problem, and then simply hand it to the UN to implement everywhere.” This was before the field invested in governance and politics; Plan A broadly similar, but with all sorts of mechanisms to fill in the gaps. As with claims back then, I think this is a mistake, and that the usual ‘muddying through’ and trial and error societies have always used remains the optimal approach, with the exception of a narrow set of risks such as biosecurity that do indeed warrant precautionary measures. There’s also something unimaginative about a world where we have superintelligence solving cancer and creating explosive growth, but that the technology solutions to solve governance problems are all top down control mechanisms rather than polycentric, competitive, or distributed institutions—which could reduce the need for a single globally coordinated regime.
This misrepresents Plan A. “Solve the technical alignment problem, and then simply hand it to the UN to implement everywhere” is not at all what we call for. In our scenario, private AI companies iteratively solve the alignment problem while regulators set a low floor by banning the most obviously dangerous AI R&D practices. Plan A is the polycentric, competitive version of AI governance. Moreover, Plan A is the most plausible case where “muddying through” works out for humanity. In the other plans, you get a few months at most to try out different alignment strategies and make mistakes before you have to hand off to your AIs and pray they’re in the basin of good deference. In Plan A, by contrast, you get to spend ten years muddling and figuring safety out on the fly.
Sixth, much of it is very unrealistic and deeply naive from a political perspective. On why China would agree to some sort of Yalta Conference 2.0, they don’t really know—they point to an appendix that says Plan will be desirable because it slows down job losses, prevents misuse, prevents WW3, avoids concentration of power and so on. So in other words: China will like it because the Plan fixes everything. What happens to Taiwan? It’s not stated, but unfortunately I suspect the answer would make Mearsheimer happy. Many other parts seem absurd on their face too, such as mandating a “heavy tax on AI persuasion.” In both the US and China, not only is everything fully transparent, but also “The public can see what regulators are doing and why.” Are they aware of how the Chinese state works? One appendix notes: “It might be helpful to use courts as a “backstop” of some kind for regulation where courts can step in if the regulator is doing a bad job”—a great example of reinventing existing structures (judicial review) from first principles. Mongolia and Canada now host data centers to enable some sort of ‘mutually assured destruction’ dynamic. What? And the rest of the world just watches peacefully? Apparently it’s ‘surprisingly easy’ for the rest of the world to get buy-in for a technological slowdown because the only three possible outcomes of AGI development are “take our jobs,” “cement hegemony forever” and “get everyone killed.” Well, no, I’m afraid (a) there are plenty of other possible positive sum futures possible; (b) even if you disagree, many actors will believe in the possibility of such futures; and (c) even within this (admittedly uncertain) verification-heavy world, many asymmetries and incentives will simply push the race into a more dangerous state-level one.
Unfortunately most of this is just name-calling, but we can restate our reasoning. The reason why China would go for Plan A is mostly that the status quo is terrible for them, and as the AIs get more capable and the US stays ahead, we expect them to wake up to this reality. Seb argues that the “heavy tax on AI persuasion” is absurd; this is an area where we actually don’t love our proposed solution. Indeed, one of my top recommendations for future work is for someone to try to operationalize this better.
Seb claims: “asymmetries and incentives will simply push the race into a more dangerous state-level one”. But this doesn’t make any sense, the race dynamics post Plan A are much better and more manageable than what would have happened by default because the countries have the machinery to slow down in the face of danger and coordinate on safer paths through the tech tree.
Seventh, their entire conceptualization of AGI is based on a very specific way of understanding the technology, how it operates, its failure modes, and its natural evolution. I sometimes characterize this as the ‘Sand God’ view of AI, which rests on a number of questionable assumptions, for example having ‘drives’, being inherently ‘misaligned’, or instrumental convergence implying an inevitable AI takeover. The authors themselves describe “godlike AIs.” While this is the dominant frame in parts of the AI safety community, it is not some sort of settled view, and rests on older theoretical assumptions that have not been developed much in recent years (since much of the field has focused on governance and advocacy rather than fundamental research). There are other ways of understanding the technology, notably Eric Drexler’s views which I particularly like. People will ask, ‘so do you not believe in ASI?’ This is missing the point: the view is not about capabilities, but about whether they must arrive packaged as a unitary entity with drives of its own. Yet these alternatives are not engaged with very much, and instead the AI safety ideological orthodoxy is simply assumed as ground truth. The same applies to how they think about and describe the dynamics of fast takeoff and recursive self-improvement, which legitimise a state of emergency; I think it’s far more likely that superintelligence neither looks like what they imagine, nor that it constitutes a sudden uncontrollable discontinuous leap. And lastly, similar critiques can be levied at the Manichean-Rationalist understanding of ‘Alignment’ as a binary state that can be ‘solved’, once and for all, in 2037: “Have the AIs been lying to us this whole time, waiting for the right moment to betray? The safety cases show that’s impossible, of course…”
What exactly are these questionable assumptions about the nature of AI? Apparently Séb thinks we can’t speak of AIs having drives or being aligned/misaligned. But then how are we supposed to speak about AI behavior? It’s not just renegade safety people who use mental language for AIs. AI practitioners within the frontier labs routinely talk about AI alignment, motivations, and so on. And they do so because it’s useful and predictive to take the intentional stance with respect to AIs.
As for our supposedly unquestioned dogmas about unitary entities with drives of its own etc… If what you mean is that there can only be one AI with one set of goals and values, that’s not what we predict. In our scenario there are many AIs with many different goals and values. But it’s unclear exactly how and why your view on AI goals differs from ours. We’ve written up our reasoning about AI goals and values here, though we believe it has been mostly obsoleted by now.
As for the “sudden uncontrollable discontinuous leap”...we don’t believe in this. Have you read AI 2027 or our takeoff model? Notice how continuous the curves are: we currently tend to expect a smooth intelligence explosion over the course of months or years, though we have heated disagreement within our team about the specifics.
Eighth, somehow in their ‘Why this much transparency?’ section, they state that the status quo today is closer to ‘Ban Open Source’ - and that their Plan A is closer to ‘Mandate Open Source’ despite them actively calling for banning open weights. There are of course important unresolved questions about open source and safety, but it seems misleading to present their plan as more permissive and open merely because they force the publication of algorithms. Not only is this a pretty bad idea from a commercial and intellectual property perspective, but it should also be less interesting to the wider public: as Epoch notes, the relative importance of algorithmic improvements has decreased over time as compute scaling accelerated around 2018. In any case, we’re now well past ‘normal’ transparency discussions in AI policy—where I agree there is a lot that could be demanded of labs. The prescriptions now explored explicitly seek to neuter commercial incentives and weaken property rights. It is also not true that “because algorithmic secrets are made public and the pace of progress isn’t accelerating, other companies will catch up to the frontier”; if your aim is diffusion (which I agree is desirable), you want to incentivize adoption and development of AI-based products and services; a ceiling as a mechanism for diffusion is like is a little like calling rent control a housing-supply policy, and compulsory disclosure will damage incentives to come up with the remaining algorithmic innovation in the first place.
First on whether Plan A is closer to mandating open source or to banning it, it helps to back up and remember that the key benefit of OS software is auditability. Instead of having to trust that some piece of software was written in the user’s interest and is not backdoored, compromised, etc, the user can just directly audit the software and confirm it does what they want it to do. In the case of AI, training code and training data is far more auditable than the resulting model weights, which are (for now) an uninterpretable pile of numbers. So if you’re a user who wants to confirm that your AI is working for you and not pursuing some covert agenda, total research transparency will do you a lot more good than mandatory open weights. Further, open weights are easier for a covert AI project or other bad actor to abuse than total research transparency, since it takes far less compute to train refusal out of a finished model than to train a helpful-only model from scratch using transparent source code. This is why we recommend TRT but not open weights.
It’s true that TRT would neuter commercial incentives to develop new AI algorithms, but this is a feature, not a bug. A key principle of Plan A is limiting algorithmic progress because it’s irreversible — once an algorithm is discovered, there’s no wiping it from everyone’s memories — and hard to keep from leaking to covert projects. RE: diffusion, we think that the natural diffusion of AIs that are as capable as the top human expert in every domain will be plenty fast enough. We think that trying to maximize capabilities and hence racing all the way to superintelligence for the sake of “diffusion” would be a terrible mistake. A key benefit of diffusion is that it will allow society to react and implement measures like those discussed in Plan A, but not if capabilities progress outpaces our reaction speed.
Ninth, a lot of governance mechanisms discussed in there are actually already things activists, non-profits, academics, and labs are working on: safety cases, third party auditing, AI control agendas, model specs, chain of thought monitoring, dangerous capability evals, and more. As Rohit writes, we already have a lot of Plan A stuff going on. These are all things people thought were never going to happen a few years ago because the normies wouldn’t wake up. And yet here we are! The same ‘state of emergency’ framing is now being applied to the next decade: the pause “gives more time for groups that don’t directly control the world’s smartest AIs to wake up to what’s happening before they lose their leverage.” I think the constant alarmism used is dangerous. A regulatory apparatus, if designed carefully, can be effective and desirable; but just as AI can be dual-use in nature, so can the dead hand of the state. It’s important to ensure that the regulatory era we’re jumping into is not one that embeds inefficiency-as-a-feature through the backdoor. But of course I’d say that—I’m an evil lab employee, after all. Alas I do think that in order to understand a technology, you need to develop it in parallel, and that trying to separate safety and capabilities in neat, exclusive categories is a mistake. We know a lot more about models today than we did a few years ago, and had we paused AI development at GPT-2, we would be in a far worse place. We do not understand everything fully, but the solution to that is to keep studying and researching the systems, not preventing their existence and reasoning about them in the abstract. Learning happens endogenously, and the scenario fails to account for adaptation outside a comprehensive emergency regime.
We see it as a pro rather than a con of Plan A that many of the requisite mechanisms are already being developed. We don’t see why Séb presents this as an objection to the plan. As for the people who thought these things were never going to happen, we don’t know any such people.
In general this paragraph consists mostly of applause lights — things that sound like wisdom but don’t actually disagree with us at all, and thereby sneakily give the author more illicit cred by making it seem like he is saying something we disagree with or haven’t thought of. Examples: “I think the constant alarmism used is dangerous.” Alarmism is by definition spreading exaggerated fears, so of course we agree alarmism is dangerous. We dispute the implication that Plan A is alarmist. “Trying to separate safety and capabilities in neat, exclusive categories is a mistake”. Again, we don’t disagree with this at all.
About pausing at GPT-2…have you read the scenario? Of course pausing at low levels of capability would reduce the rate of safety progress relative to pausing at higher capability levels. This is why we call for continued scaling to ~max controllable capabilities level before pausing. You can see our more detailed analysis of the pros and cons of scaling faster or slower here.
Conclusion
Overall, Séb’s piece is mostly inaccurate and fundamentally misunderstands what we are suggesting. It seems like in practice, Séb is advocating for something like Plan D, i.e., what happened in the race ending of AI 2027. We’ve presented many arguments for why we think this is terrible — it would pose unacceptably high AI takeover risk and massively concentrate power into AI companies or the US government.
There are some real problems Séb points out (e.g. our proposal to limit AI persuasion), but he doesn’t seem to acknowledge that these problems exist in his preferred world as well, except they are much worse because there is no slowdown and less transparency.
Still, for this reason and many others, Plan A is far from perfect. We wish that we had more specific and better proposals for many parts of Plan A, and welcome suggestions for improvements.
- ^
Nvidia is currently making large profits, and in Plan A we expect that to continue. We do think the AI companies will be the single biggest winners in an economy transformed by AI, capturing much more of the profits than, eg, workers or owners of capital other than semiconductors and robots.
- ^
Our policies—most notably (i) the massive AI capabilities slowdown and (ii) total research transparency—would dramatically cut AI company profits relative to the Plan D counterfactual in the short run. In the slightly longer run, profits are much greater than they would’ve been in Plan D because we would probably be dead.
- ^
The AI companies do not get nationalized in Plan A, and in fact the government’s ability to bully them goes down compared to today thanks to the transparency. (For example today the exec branch can decide to export control models, preventing them from being deployed externally, and the public isn’t able to see how the decision was made or whether the model in question really was more dangerous than competitor models for example.)
- ^
In the short run, that is. In the slightly longer run, unless we get lucky with alignment, company profits fall to zero in Plan D when everyone dies.
Some of these do seem like misrepresentations. For example, about your beliefs regarding to whom the economic benefits of AI will accrue, and also about the desirability of having a central planner.
Others seem mainly like disagreements about what the likely effect of proposed policies would be and/or qualitative judgments about those policies.
For example, it’s possible that mandating transparency on AI labs gives more room for policymakers to iterate, but it’s also a very big upfront change. It’s plausible someone could consider that change to bake in too much upfront, without misrepresenting the plan.
It’s also possible that the existing powers the government has are comparable to or greater than the various powers you propose, so that it’s no big deal to give them more powers. Others might disagree. This also seems more like an area of policy disagreement rather than misrepresentation.
I might have misunderstood which are misrepresentations and which are disagreements. Either way, I would be interested in some clarity as to which of these matters you vehemently disagree with Séb on and which you feel are misrepresentations.
Thanks for this comment!
I think the central disagreement is about whether AGI/ASI is real or not, and almost everything else is just downstream of that. The claims and policies made in AI 2040 only make sense in a world where you can have AIs that are much much smarter than humans, and that mere AGIs can cause explosive growth.
Given that Seb disagrees with the economics substantially, I would guess that he thinks that AIs will never reach the point where they can fully automate the robot and semiconductor supply chain; if we do get full automation then fast exponential growth seems to quickly fall out of reasonable modeling. Unfortunately I don’t understand his views that well, so it’s hard for me to say exactly where he thinks AI capabilities will cap out.
Re: Transparency and government powers, I think his post pretty badly misrepresented what we were saying, where I think someone who read his piece but not ours would be extremely misled about what sorts of regulations we were proposing. But I also imagine there’s substantial policy disagreement there… so from my perspective it seems like both?
I will say that with government powers in particular, Plan A does involve governments into AI more than the status quo. Insofar as you think that’ll be predictably bad, that is a genuine downside of Plan A. There’s a sense in which any regulation is “central planning”… and yes, we do advocate for certain types of AI regulation. But overall in Plan A we aim for as market based and decentralized solutions: for example, we propose several cap and trade regimes, which seem to me the maximally libertarian way to go about this regulation, and so calling it “central planning” seems pretty misleading. Other regulation is more difficult to set up this way, such as limitations on algorithmic progress.
Thanks for this reply. I do agree that many people who are skeptical of various AI safety proposals don’t really believe in AGI or ASI, but I know that’s not everyone. For example, I am skeptical of AI 2040’s proposals but I very much believe in AGI and ASI, and that’s at the heart of some of the reasons for my concerns. I don’t know if that’s true in Séb’s case: is there a particular thing he wrote which makes you think he does not, or are you speculating/generalizing from experience with others?
Regarding the misrepresentations vs. disagreements, what I would really appreciate is understanding which are which in this post. I count 16 quotes from Séb listed under “False Representations”, but some of them seem more like disagreements to me. It would be helpful if you could clarify which you believe are actual misrepresentations and which you believe are disagreements.
I think it’s true that Séb doesn’t believe in AGI/ASI in the sense that I mean: see his fourth response here. Also see e.g. this model he posted which argues that comparative advantage implies postASI humans will still have jobs (which doesn’t make sense if you think ASI can use the inputs that humans require much more efficiently than the humans do).
Seems very reasonable to be skeptical of many of AI 2040s proposals despite believing in AGI/ASI. I’d be curious to hear more (though perhaps you should post those on the main post, not here, if you feel interested :) ).
Re clarifing the quotes; reading them now I still think they are all misleading but to varying degrees and probably won’t spend more time further getting into it because there’s a lot on my plate right now, but if others think it’d be useful for me or someone else at AIFP to do another pass here trying to further clarify let us know (e.g. by reacting or commenting here).
I don’t think belief in ASI necessarily implies belief in all of:
> (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
Or did you mean some other part of his fourth response?
I also don’t think it requires you believe that no comparative advantage for humans post-ASI exists: positional goods are still a thing, a price premium for human-produced goods is very plausible (and positing that it won’t happen is not a question of capabilities).
I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.
As I said in the post, this part is largely a strawman of our views. The part I mostly meant was “Nor do I think we will get real GDP growth of 50% in 2032” and “the model moves far too quickly from AIs being able to perform tasks to robots being reliable, legally deployable, organisationally integrated substitutes for almost all labour, and from there to a closed-loop reproduction of capital”. Perhaps what he means is that there will be delays, but eventually we’ll get 50% GDP growth?
I tend to find the exact opposite is true. People love claiming that the real disagreement is real world bottlenecks, but usually its really differences in capability expectations. I think the AIs will be wildly superintelligent, with vast quantities operating at the equivalent of 100x or 1000x human speeds, also with a much higher qualitative intelligence due to things like knowing way more than any human can know, having much more experience than any human can every get, and having a physically much larger and more connected brain such that they are able to make discoveries and model things that are far too complicated for humans.
This is obviously an extreme milestone; we outline earlier milestones here: https://ai-rates-calculator.vercel.app/, for example. I would encourage people to make forecasts of when they think these specific milestones will be crossed (which might be “in hundreds of years or never”). If earlier than that, maybe its just a timelines / takeoff speed disagreement.
I kind of expect that many people would argue that this is the wrong ontology for thinking about AI capability progressions, and that actually, it’s somehow going to be more diffuse/multipolar or something, in a way where thinking in these terms isn’t useful for modeling the world, in which case I’d love to hear a better frame.
Still, I think that most people with very different views would tend to disagree that the notion of ” superintelligent” AI that I outlined above would happen but will have small effects on the world. I think this view is extremely hard to make coherent. Once we’ve got AIs like that, the cognitive labour supply would become enourmous, such that almost all the cognitive labour happening would be AIs, not humans.
Totally appreciate you guys have a lot on your plate right now! At least two of the examples I mentioned in my top-level comment didn’t seem like misrepresentations to me (though two of them did), which makes me confused as to how much of this post is accurate if it really is saying that all of the quoted examples are misrepresentations (hence the reason for my comment requesting that clarification). I haven’t carefully dug into every claim myself but overall my sense is that this post very heavily blurs the line between vehement disagreement and actual misrepresentation such that I don’t feel there’s a clear takeaway for me about how much of each is going on in Séb’s critique.
If I were trying to do this today and I had the choice, I would take the weights and not the data + algorithms. We’ve a reasonable collection of interp techniques that demand weights only, and a much smaller collection of techniques that demands data + training code only. I may be wrong, data + code is certainly helpful and both together is clearly better that either alone, but that’s my best guess.
(assuming I can’t actually go ahead and train a model on a significant fraction of that data)
Thanks! My intuition is the opposite, but I’d be curious to know which interp techniques you think are most valuable right now.
Though actually I realized that paragraph in the post is somewhat misleading (in a way which strengthens your point), sorry about that! Total research transparency gives you ~all the code, and some of the training data, but most of the training data is in the opaque database, and so can’t be fully audited except via AIs, which might still get you a bunch of the benefit. (Read our detailed proposal here: https://ai-2040.com/supplements/transparency-plan. )
So this is a guess, but:
I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning
Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it
Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data
Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://arxiv.org/abs/2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).
Given data/code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too
I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag
Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.
I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task
(edit: this opinion no longer seems so unanimous)
You miss the point. Suppose that Claude Bookshelf 7 and DeepSeekv7 have the same capabilities, and Claude Bookshelf 7 has the entire training run audited and DeepSeekv7 is open-sourced. Then DeepSeekv7 gets finetuned by terrorists, but safetyists fail to evaluate it for reasons similar to the last paragraph of METR’s report on GPT-5.6 Sol: “As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems.”
I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.
I’m mostly convinced by Plan A (given the techno-economic assumptions).
I still struggle with TRT as plausible without radical internationalisation. Financial incentives are too strong for companies, geoeconomic incentives too strong for states. The social contract would likely be unstable and therefore hard to enforce. AI companies already have strong levers with governments and are likely to use them to obstruct, dodge, cheat or FUD. We don’t have enough time for it to become incontrovertible that cigarette smoke causes cancer.
I would thus trade algorithmic governance (inspired by Bioengeneering) with some open weight release (for Unis, independent researchers to participate in alignment research for positive long tail spillovers), ie (some) Research Transparency with (some) open weights.
You might be interested in other proposals we make such as filtered transparency (see here). I don’t fully understand your comment but I do think that TRT is a bigger lift than filtered transparency. We recommended it because I think it is the best option, and I think it’s viable if enough people push for it. But TRT is NOT a load bearing part of Plan A; more lax transparency proposals are very much consistent with it.
Hi Thomas, sorry this was not very clear,
I was trying to paint a rough sketch of “spillovers in context” most of which is done well in your scenario planning. My disagreement largely with the current rapidly developing “state-AI complex”. The amount of private (and likely soon, public) capital, talent, state interest concentrated make it unlikely that TRT will be acceptable.
I would argue AI2040′s linchpin is compute. Coordinate compute, open up almost everything else. Governments and companies want control of many other things, algorithms, talent, data etc. These will also be negotiable between companies and states. This means companies, states get to keep certain advantages in return for offers . Algorithms alongside compute are likely to be the most significant assets under negotiation & both will be part of asks and offers.
I will definitely have a look at filtered transparency, it may help anticipate some of the mechanics here.
I agree with this in the current regime, however as RL becomes an increasingly large share of training compute, I think it becomes increasingly important to access model weights.
Rollouts are a good starting point—though maybe these aren’t even counted as “training data”, since they can’t be audited before training—but they don’t really address concerns about how the RL generalized off-distribution, whether it selected for hidden reasoning/scheming, etc. In this setting, the actual policy and reward models would probably be more informative for auditing than just the training code and data.
This response is too angry and generally not a good look. I don’t have a position on the object level at the moment, but reading some of this response makes me think more favorably of Séb; a lot of what you say seems like disingenuous nitpicking and yelling “we didn’t say that!!!” when you said something quite similar and it’s clear Séb simply disagrees.
For example, you say:
This is kind of ridiculous: you believe (b) not all profits accrue to the labs, because some accrue to NVIDIA and a tiny handful of other firms. Séb disagrees, clearly. But instead of addressing your disagreement productively you accuse him of lying. (c) You do believe that Plan A would not significantly decrease the relative profits, i.e. you think that even with the extreme slowdown, the AI labs eat the world. Séb disagrees. (e) I do think Plan A sounds like de facto nationalization, and I think it’s disingenuous to imply that the government can already do more (it costs the government a large amount of political capital to try and they may also lose in court).
I’m disappointed by this response.