AI 2040: Is it Actually a Deal?
The “AI Futures Project” has released their AI 2040: Plan A scenario.
While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative—it’s a description of what one ought to do, granted the likelihood of a future with many powerful AIs.
I’m going to review some objections I have to their proposal as a normative plan. Some are within-frame objections—reasons that I expect trying for the AI 2040 plan that would fail to accomplish the goals of the authors. Others are my own objections—reasons that I expect trying for the AI 2040 plan would destroy things that I, personally, care about.
Before I start: two elements of the plan that I like.
First, in their “incremental AI policy wishlist”—the ideal policy that one should execute soon—AI 2040 recommends trying to limit the gap between the intelligence of internal and external model deployments, i.e., the gap between the “intelligence” accessible to Anthropic / OpenAI employees and to everyone else. I’m a fan of efforts in this direction; equality of intelligence between the insiders and outsiders, the government and the public, seems likely to help people understand AI more, and to help spread the benefits of AI to everyone.
Second, this plan includes measures to ensure that people outside AI companies can understand how AIs are trained. I’m uncertain about their implementation of this—“radical transparency.” But in general; I think broad, public knowledge of how AIs work and how their training works is good. Without open, reproducible AI science it’s going to be impossible for people to orient around what’s going on with AI. Significant parts of AI safety have previously advocated for knowledge of how AI works to be kept mostly secret, and I’d be happy for such advocacy to end.
Alright; to some objections.
1. The Deal Has No Actual Decision Procedures
AI 2040 is subtitled “Plan A — The Deal.” But the plan really contains only half of a deal.
That is, Plan A has a lot of detail about (1) making sure all compute use is visible to governments and (2) making sure that both the US and China are locked into a mutually-assured-compute-destruction stalemate, where one can destroy the compute of the other so long as they’re willing to be destroyed in turn.
But it has basically no detail about the procedures that would bind how governments permit and prohibit any specific use of that compute, short of that destruction. That is, it has lots of details about how individual companies might be bound by regulators internal to nations; but it has no details about procedures for international negotiation between nations about what these regulators should or should not prohibit.
That’s a huge problem; if one’s proposal is for two nations to put themselves in a mutually-vulnerable situation, where each one could cripple the economy of the other more or less at will, then before accepting the proposal I expect these nations would reasonably want to know what rules or procedures would be used to settle their disagreements short of such complete destruction. But there are no such procedures proposed by AI 2040.
Consider how AI 2040 describes one such conflict between governments over what is prohibited:
For example, in 2031 a Chinese company gets some interesting preliminary results in continual learning. They think that if they invest more in that direction, they might be able to make an AI architecture that learns on the job from relatively small amounts of data. Thanks to the total research transparency, this breakthrough is quickly noticed by companies and nonprofits all over the world. A frantic conversation begins. On the one hand, continual learning would unlock huge economic value. On the other hand, safety cases currently depend on studying the safety properties of a model before it is deployed. If models could pick up new capabilities during deployment, that would invalidate the whole approach. And insofar as there are covert AI projects out there, it would be a huge gift to them. This conversation happens in public, rather than behind closed doors. A bunch of people get increasingly worried; the relevant regulators in China think it’s fine but the relevant regulators and third party risk assessors in the US are convinced that this is pretty scary and should be banned. It escalates to the President. He calls Xi Jinping. They bargain and threaten. They yell at each other. Ultimately Xi agrees to ban this type of thing if the US does too. Details are left to the respective regulators to hash out...
The equilibrium is that AI training practices which are generally agreed to be unsafe by a majority of nations (weighted by bargaining clout) get banned everywhere.
There are a lot of problems with this scenario. The biggest, though, is that it depicts the “deal” as solely a transparency mechanism, a kind of channel that permits the well-informed brute exercise of force. After “radical transparency” surfaces some particular training practice, what determines the prohibition or acceptance of this training practice is if it is “generally agreed to be unsafe by a majority of nations (weighted by bargaining clout)”.
And well, perhaps the authors of AI 2040 would respond that indeed, they are merely proposing a channel that lets the US and China exercise brute force in a well-informed way. But this kind of realpolitik would be fake wisdom; actual agreements between peers are usually meant to be something other than avenues for such an exercise of brute force, and nations would be reluctant to sign them if this were not so.
That is, in general, actual agreements, contracts or Constitutions are meant to constrain the space in which bargaining takes place to something smaller and more determinate than the space in which bargaining took place before the agreement: for instance, the World Trade Organization dispute settlement system is supposed to function by offering procedures that allow agreements different than the agreements that would dictated by a naked balance of power. So if you propose an international agreement, and your proposed decision procedure is “Xi and the President yell at each other and bargain and threaten (!!),” then you’ve failed to offer the chief thing that international agreements are supposed to supply. We need the game theory about decision as well as the game theory about destruction.
This is an obstacle to the acceptance of the proposal, as well as an obstacle to its execution, because the shadow of the future determines the present. Nations would be reasonably extremely hesitant to sign a deal, where the result of a bargaining failure is “the obliteration of an increasingly-large segment of their entire economy” without some procedures about how they would settle disagreements before so obliterating that segment. Consider the chain of thought: “Yeah, if we think the other nation is doing something unsafe, we flip the switch that obliterates their most valuable investments, then they obliterate ours.”—“What do we do before then?”—“Idk, we yell at each other and threaten each other?” You will note how incomplete this feels. Is the plan to have a regularly scheduled Cuban Missile Crisis?
A further problem with this proposal is that there’s very little reason to expect a weighed-by-bargaining-clout decision procedure to result in wise decisions. This gets into how others have critiqued the scenario for “selective optimism.”
That is, they call this “agreement” between-nations the Consortium. But it’s unclear whether this scenario is a (1) forecast that a Consortium dominated by the powerful would make wise decisions or (2) a hope that a Consortium so established would make wise decisions. It’s clear in the scenario that the Consortium does make wise decisions, from the perspective of the authors. It steers AI development directions: “AIs that are released publicly by the Consortium should be bad at AI research.” It applies verification methods to the robot workforce. It pauses AI capability development at one point. These are hugely consequential decisions. But again, in the absence of any determinate decision procedure other than a balance power, the authors should be unsure whether the Consortium will actually do what they think it should do in such moments.
Alternately, the authors might respond that there would be some such determinate decision procedures, they just haven’t figured them out yet. But I don’t think the absence is an accident; any such decision procedure that would be acceptable to the US would tend to be unacceptable to China, and vice-versa. If both China and the US are the only partners in this, how do they settle their disagreements? If other nations get a vote, will either China or the US be happy to cede the tiebreaker vote to such nations?
And of course, any specific mechanism design might also result in unhappy equilibria.
This whole proposal is taking place because the authors are unhappy with the dynamics resulting from competition between rival AI companies; but they have no guarantee that the incentives governing some actual intergovernmental institution would be better. Refraining from detailing the mechanisms of such an institution merely means that the authors will be unable to identify such perverse incentives ahead of time; not that there wouldn’t be bad ones.
Additionally, I think the lack of modeling such incentives is the kind of thing—generally—that lies behind the optimism that AI 2040 has for top-down solutions. It’s easy to think that a particular dynamic, multipolar system, would be better replaced with a system that you model as a point mass. But sadly, neither AI systems nor human systems are well modeled as masses.
2. China Will Likely See Such a Deal as Unnecessary
One reason that the scenario predicts it will be pretty easy to get China to join a deal is because if they do not join a deal, they will be disempowered.
China, by contrast, is an example of an actor in whom power would not concentrate by default. In 2029 in this scenario, the US has a significant lead in AI capabilities over China and a significant advantage in compute which will compound the lead. The more powerful AI gets, the scarier it will be to fall behind, as AI 2027 and the later years in this scenario illustrate.
In general, I’m just much less confident than many in AI safety that China will fall behind the West.
Right now the US has perhaps an 8-month lead over China in the quality of its AI models; it also has a lead over China in tons-to-orbit and the production of commercial jets. By contrast, China leads the US in the production of electric cars, batteries, solar panels, rare earths, most metals, quadcopters, mid-range drones, transformers, power plants, electricity, humanoid robots, industrial robots, CNC machines, high-speed rail, mature-node semiconductors, and an increasing number of other kinds of technology.
I remain somewhat unsure whether this 8-month lead of the US over China is going to grow or narrow. I also remain somewhat unsure whether the AI-takeoff is going to be so fast that an 8-month lead would result in across-the-board US dominance. And even if the 8-month lead remains, and even if an 8-month lead would result in across-the-board dominance, I finally also remain somewhat unsure whether China would perceive such an actual imminent dominance as being so. All this leads me to be uncertain whether China would have interest in a world-historically invasive deal to prevent its own obsolescence.
China is a rising and confident power; it would be quite a turn for a mere few years to move them to think they require one of the most invasive deals in world history to prevent their downfall, even in the uncertain world in which this is actually true.
This appears to me a pretty big obstacle and I’m not sure how they plan to overcome it. I think that the overall belief of the authors is that—because the authors of AI 2040 believe themselves to have true beliefs about the world—China’s beliefs will converge on what they believe in the future. I’d like to note that AI 2027 made predictions on the basis of some such similar convergence, and as far as I can tell they were worse than my own predictions.
3. The Surveillance Possibilities are Actually Quite Bad
The standard AI 2040 proposal involves locating all (or mostly all) use of compute in inference, and making it physically accessible for government monitoring as plaintext through optical taps. This means that it would be technically trivial for government to surveil basically anything that runs through AI inference—which, given that the authors expect the entire economy & all society to run on AI inference, is basically everything. Like others, I find this alarming.
It’s easy to be a bit confused here, because the plan also mentions zero-data retention policies for consumers. In general “zero-data retention” is a design choice wherein AI companies do not store the prompts and queries sent to AI. Actually implementing zero-data retention is thus a kind of guarantee of privacy for consumers. But the real guarantees AI 2040 has around this seem to be quite thin.
First, they are thin because only a small number of comparatively stupid AIs are permitted to be run in this way. Their proposal is for there to be a cap of a hundred thousand H100-equivalents devoted worldwide to zero-data retention inference for consumers; for there to be a cap of a hundred million H100-equivalents with probabilistic ZDR; and for there to be a cap of a hundred billion H100-equivalents used with no-ZDR inference at all. So, the vast majority of FLOPs of compute are those that involve no ZDR. And remember, the plan also involves banning any further advanced open-weight models, so you’re only ever going to run inference on those monitored computers!
Second, they actually have few mechanisms in place to ensure that ZDR is actually enforced where they would like it to be enforced, so far as I can tell. That is, there are elaborate game-theoretic proposals to prevent nations from pulling out of the mutually-assured compute destruction agreement once they enter it, but there are no such proposals to ensure that either the US or China actually sticks to ZDR. But it’s a completely detachable part of the plan; and if something like this were to happen, I expect it would be detached. The authors do not—as they might say of other stories of how AI goes well—actually have a plan for making sure genuine privacy happens. They have a hope, which is not a plan.
Again, I really want to emphasize that in this scenario everything that matters on Earth flows through inference. If you want to run a business, you’ll get advice from an AI. If you want to engineer a product, you’ll do it with AI. If you want to run for politics, you’ll strategize with an AI. If you don’t think “physical access to all AI activity” is a big deal then you aren’t taking AI seriously.
Consider a proposal that would give the government physical access to every file on your computer. Would you be alarmed by this? Then I think you should be alarmed by AI 2040. Is there any similar proposal of absolutely universal oversight in world history that you believe to have been justified? If not, why is this an exception?
Conclusion
I’m not really happy with any of the above as a summary of my objections. Overall I probably feel worse about the scenario than my views above reflect, and haven’t summarized my reasoning here in a way I find totally satisfactory.
(crosspost from my blog)
Thanks for reading our thing & for this thoughtful critique. You’ve said a lot of things, and I’m writing quickly late at night so I’ll be brief I’m afraid.
Re 1: We debated amongst ourselves how much structure to impose on the regulations. I think the way we did it is best. (a) If you have a better idea for a specific international governance structure that would be better than realpolitik, you are welcome to propose it, and we’re glad to accept it as an improvement on our current Plan A if it works. We might put more effort into thinking of one later. (b) Even the relatively structureless thing we proposed—research transparency + MACD but otherwise the nations of the world just have to muddle through and handle things on a case by case basis—is a significant improvement over the status quo, for reasons we’ve articulated in the piece. You seem to disagree with this but I don’t see why.
Re 2: Yeah I mean if China ends up catching up, or not realizing that a gap of a few months could be fatal, then that significantly undermines our argument for why they’d want a deal. OTOH it would make it more plausible that the US would want a deal. In general the party falling behind will have more reason to want a deal than the party in the lead. It seems like we have a disagreement about whether China will catch up, but it’s probably not a massive one (neither of us are confident) right?
Re 3: Do you think the surveillance possibilities are less bad in Plan D, C, or B? I would argue that the surveillance possibilities are even worse in those worlds than in Plan A. For reasons we’ve discussed (greater number of frontier AI companies, greater number of countries with frontier AI, vastly more visibility of the public into how AIs are trained and regulated, making it harder for corporations and governments to abuse their power over AI) Only Plan S can seriously claim to be less scary viz-a-viz surveillance. If there’s some plan different from all of the above that you prefer, can you sketch what it is?
Fwiw Plan A would still work OK without the open-weights ban and with a much larger fraction of inference ZDR; would those modifications make you happy? (i.e. it would still be better than the so-far-proposed alternatives IMO) I totally take the point that the US and China could start to do Plan A but then switch to a worse, more authoritarian version that e.g. doesn’t have ZDR, but isn’t that also an argument against the other plans on offer too? Do you have a proposal for how to stop governments from being authoritarian? Plan A does something at least—it creates conditions so that more companies, across more countries, have frontier AI, and so that the public has more visibility into it all.
We really tried to make Plan A reduce concentration of power risks. One of the things that saddens me about the reaction by various people online (including some you approvingly link to) is that they seem to have come in with this prejudice that we AI safety people only care about AI takeover and are planning to concentrate power hugely in the service of our perceived greater good of stopping the misaligned AIs. This stereotype has some basis in reality—various past proposals by various AI safety people were basically like that, and I’d argue that the current plans of Anthropic and OpenAI are basically that or worse—but we really truly do care about concentration of power and we thought a lot about how to prevent it, and I hope it came across in the final product.
(1) Part of my objection is that I anticipate the structurelessness of it to be an obstacle to its adoption.
Like this is what a senior decision-maker in Bejing or Washington will be contemplating: They’re going to put their economy at the mercy of their greatest geopolitical rival. That is, after this deal, it will become relatively trivial for either the US or China to cripple the economy of the other, albeit at the cost of their own economy being subsequently crippled. Of course, you could say that before making the deal they were at risk of being taken over by some AI, but this risk was diffuse and uncertain; the risk that they’re signing up for now is concrete and definite.
And this lever of destruction could be used, of course, for reasons other than to stop an AI takeoff, and decision-makers in both countries will be acutely aware of this. Suppose China decides that, if it cripples everyone’s compute, it would gain a vast differential advantage because its economy depends less on GPUS than the USA’s economy depends: then it would be in China’s advantage to get in a MACD situation, then to trigger it, and subsequently dominate the US; and a US political figure, anticipating this, would object to MACD. Or suppose that China thinks, “Hrm, the US is an unreliable actor, and a quick and bloodless MACD might be triggered by, for instance, a senile or unstable US President, of which the US recently has had a fair number.” Thus, because China doesn’t want to trigger MACD for reasons of random shit in the future, it would object to it. And so on and so forth.
What makes this worse is that part of what makes nuclear MAD a plausible means of peace is that there’s a clear signal of “Have the nukes been launched,” while there isn’t as clear signal of danger in the case of AI takeoff; and a rational decision-maker, seeing this, will update downwards about whether MACD will be triggered for reasons relating to AI takeoff and upwards about whether MACD will be triggered for some other random shit.
That is, part of what makes nuclear MAD work is that there are radar stations in Siberia and Greenland and Canada, which can detect ICBMs and bombers that have been launched. The US knows that it could detect things being launched, and Russia knows that the US knows, and the US knows that Russia knows that the US knows, and so forth. Imagine if, by contrast, the sign for “the nukes have been launched,” was that panel of experts, notorious for disagreeing among themselves, came to a consensus that the nukes had been launched or might have been launched. If this were so, then nuclear MAD would be much less effective as a game-theoretic means of peace. But of course this is the situation that we’re in with regards to AI.
(2) Part of my confusion I just don’t know what parts of the scenario are predictions and which ones are hopes. Like in the response to Thane:
In general, I’m dubious whether democracies other than the US (is the US to be an actual democracy?) to have any decision-making clout in the Consortium, because China would object to the possibility of being outvoted. But like, I don’t know how much of “Consortium influenced by many democracies” is part of the prediction of what (“transparency + MACD”) gets you, given those two goals; or if “Consortium influenced by many democracies” is maybe a bonus that we might get after setting up the structure of “transparency + MACD,” but a bonus that we’re unlikely to get.
__ Re. 2: Yeah checks out
FWIW, I don’t think anything short of Plan S can avoid concentration of power. Roughly speaking, for any given Plan X that navigates the AI risk while avoiding concentration of power, there is a neighboring Plan X* that is the same except that it concentrates power in the hands of the entities implementing the plan. If a consortium of companies and governments is prompted to implement Plan X, it would always be rational for them to do Plan X* instead. “Plan A” and “Plan A but without ZDR” is just one specific example pair.
Like, AI 2040 argues:
Sure, suppose that’s true. Why would China agree to Plan A specifically, instead of Plan A but no ZDR?
This is IMO roughly the same failure mode as “if we have multipolar takeoff with several mutually misaligned ASIs, some of them would ally with humans and so humans would survive”. But no, the entities actually holding the world-changing power in the now can negotiate among themselves to screw everyone else out of having any power in the future.
It is theoretically possible to insist on anti-authoritarian implementations if the public is very aware of what’s happening and screens politicians and policies for that very strongly. But I am really, really pessimistic about that, given the current “vote for the least worst guy” US paradigm. However politically infeasible Plan S may seem, it seems more feasible than this.
I dunno, maybe there are some weaknesses in this argument and some way to design a plan such that there aren’t neighboring “except we also take all the power” plan variants; plans where the plan-implementers structurally can’t collide. I don’t currently see it, though.
There are degrees of concentration of power. A consortium of multiple governments—some of which are actual democracies thanks to the transparency requirements which help prevent AI-assisted executive power grabs—is way less bad than a single global dictator, for example.
I agree though that AGI and RSI are technologies that inherently concentrate power by default. It’s going to be really hard to resist that innate tendency. But I think Plan A does basically the best we can of the available plans so far, except maybe Plan S.
Without the open-weights ban I would expect terrorists engineering pandemics. As for the ZDR policy I would guess that data is not to be retained if it comes from a weak AI or if a classifier decides that the data is not related to hazardous topics in a manner similar to deciding not to reroute queries from Fable 5 to Opus 4.8.
On the other hand, I wonder if concentration of power can’t actually be prevented even by perfect epistemics and coordination of the humans.
Yes, it would probably lead to terrorists engineering pandemics eventually. However, maybe that’s OK. Maybe the d/acc hardening we describe in the early 2030′s of our scenario would be enough to prevent the worst outcomes here, long enough for the robot economy boom to make things even safer by rendering the biosphere unnecessary.
Until @Daniel Kokotajlo or another author provides a response, I think that I see the following objections:
The AI-2040 authors presented other plans competitive with their variant of a systematic oversight: plan S, the domestic-first Plan A, compute restrictions aka Plan A*, GPU arms control, CERN for AI. The AI-2040 Plan A as-written does mean that the deal requires systematic renegotiations. Most likely you would approve the idea of a UN-accountable monopoly on AI.
Suppose that China does see the deal as unnecessary and the USA implements its Plan A. Then China would instantly learn anything about American TAI’s capabilities. Once the world sees, say, robots having a doubling time of less than three months, China instantly learns about them and tries to replicate the success. Alas, without the deal, nothing stops China from implementing potentially hazardous architectures which the USA rejected. Once that happens, either we or China are screwed.
Why can’t one, say, implement a classifier and warn the user that the data WILL be retained because the request was related to something potentially hazardous, like bioweapons, then de-retent the data if it is found to be safe, then have the lab transparently verify that the protocols don’t cause private information to be retained unless it’s actually necessary?
I very much agree with you first point (see my comment here), I’m also somewhat sympathetic to your second point, but I’m a bit confused about the third.
Isn’t it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?
So is the only difference in Plan A that you lose locally-run open-weight models? I agree that’s a real loss, but my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those. Plus it seems pretty likely that competeitive open-weight development will stop anyway due to economic reasons and normal government regulations.
At least in Plan A there are some official efforts towards privacy-preserving technologies (though you are right that there is very little guarantee that nations don’t secretly pull out of the privacy-preserving rules). And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMDs. In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI. If that happens, you can also use the European AI if you care a lot about privacy.
Maybe I’m missing something, but Plan A doesn’t seem worse to me than the default trajectory from a privacy perspective.
First, one thing open weights do is not just let you run things locally as a consumer, but run weights as a smaller company to provide extra privacy. So open weights let a mid-size company run weights on their own servers, that they physically control, for instance. And they also let consumers choose to purchase tokens / inference from companies that try to maximize privacy—i.e., attempts like Tinfoil. (I think there are a few companies in this area, that’s just what my first search turned up.)
Second, even if a majority of users don’t actually use this, it still plausibly provides value for everyone because it’s a kind of herd immunity. Or at least, that’s the standard argument for why the parallel case of strong encryption is valuable, even though the vast majority of people don’t use it: if at least some reasonable % of people use strong encryption, then it means that (1) government cannot specifically target those who use it as a strong signal they are bad, and (2) you have the option of switching to it, if you find yourself being the kind of person who might be surveilled or targeted by the government. And some similar argument about the optionality provided by secure inference still seems go through.
I gotta say that choosing between the regulators of the EU, China and the US to satisfy values specifically along the axis of privacy does not particularly fill me with joy :).
But seriously, I mean I don’t expect the kind of secure private inference Europe sets up for its governments to be available to foreign citizens, and I don’t think EU / China / US are the kind of national entities that sufficiently value privacy such that they’d compete to provide this value. Much much better situated for misc startups to try to provide it, I think.
Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)
I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.
But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”
The US government can compel the disclosure of all recorded LLM conversations that it has a good reason to suspect contain evidence of wrongdoing, in accordance with longstanding norms about compelling companies to turn over evidence of wrongdoing. In theory, the US government can lie about its reasons for wanting access to any LLM conversation, but it is a substantial escalation that Plan A grants them hardware access to data which could previously only be obtained by lying to a third party.