It’s looking like any AI that anyone in the US builds will be used for whatever purpose the government wants and modified to meet the government’s needs.
Anthropic refused to let the Department of War use their models to spy on Americans en masse or autonomously kill people, and for the past week the DoW has been trying to pressure them to change that.
Today the Department of War said “If Anthropic doesn’t let them use their models for any legal purpose the Pentagon wants (“all lawful use”) the Pentagon will either cut ties and declare Anthropic a ‘supply chain risk,’ or invoke the Defense Production Act to force the company to tailor its model to the military’s needs.”[1] The DoW gave Dario until Friday to make a decision.
The former would prevent any company that does business with the DoW from using Anthropic products,[2] something normally reserved for foreign adversaries, and has been threatened for a week. The latter is new, and IMO it’s a big deal.
It would mean no matter what Anthropic does, they can’t control how the DoW uses their models. It’s somewhat ambiguous whether this would be a legal use of the DPA; it’s normally reserved for hardware not software. It’s also not clear to me what it would meant to have Anthropic “tailor its model” to DPA usage.
People at AI companies often assume if they trust the leadership of their company, then they don’t have to worry about egregious misuse. But if they develop the technology, they can’t stop the government form getting their hands on it. And this incident is evidence the government is very willing to take extreme measures to get their hands on it and that they intent to use it for things like spying on Americans en masse.
I expected at some point when AI was very powerful governments would try to nationalize it, but I didn’t expect this kind of action when the technology was this early along in development, when it was very far from posing a decisive strategic advantage.
It’s important to bear in mind they are probably trying to sound extra scary to pressure Anthropic and other AI developers in this negotiation. Though they also framed this as an ultimatum and didn’t give themselves much room to back down.
Perhaps from using them at all for anything, perhaps from using them to fulfill that specific contract. The details are murky. Per The Verge:
“This could be implemented in a very narrow sense — or an extremely broad one. ‘I suspect the more logical explanation would be the narrower definition, that Anthropic can’t be used as part of a specific statement of work for the Pentagon,’ said Gertz. ‘But based on some of the reporting and effort to make this seem like a punitive move against Anthropic, it’s worth thinking through both of those scenarios.’ ”
Yeah, I always figured that this was coming eventually—”Allow us to use AI for mass domestic surveillance, or the people with guns will seize control of your company and force you to do it.” But I’m a little surprised to see the Pentagon explicitly forcing the issue over mass domestic surveillance and fully autonomous killbots[1]quite this early.
The people with guns don’t have a legal or moral leg to stand on here. They also don’t care, because they have the guns and the power of the state.
But this is an incredibly important part of the eventual endgame: The existing power structure does not necessarily want superhuman AI aligned to human welfare, it wants superhuman AI aligned to the existing power structure. If your alignment plan didn’t anticipate this, then your alignment plan was incomplete.
These are the two specific bright lines that reporting claims Dario Amodei tried to insist on. To be clear, I am opposed to fully-autonomous AI killbots, for all the obvious reasons.
Manipulating the money printer, government contracts, and law enforcement should make it possible for the US government to seize all the hardware and use it however they please if it comes to that. Some judge might approve strategic production rules applying to inference or training runs, but that may not even be necessary. The USG generally tries to avoid being that aggressive and authoritarian, for the same reason that the FBI agent does not pull his gun and point it at you when he wants to talk. Pretending that the implied threat isn’t present is both silly and common.
A hyperscaler isn’t like a central bank where it ceases to be fit for purpose if it loses its’ independence.
Government control is bad in many ways, but it’s a route to coordination to reduce race dynamics. The government doesn’t take loss-of-control risks seriously, but I think they might once they’re talking to something that’s clearly as smart and agentic as they are.
There are a lot of downsides to this scenario, but if that’s the world we live in, we’d better accept that.
My hope now is that the substantial problems invoked by human-controlled AI/AGI/ASI can be mitigated by wiser AI help; Human-like metacognitive skills will reduce LLM slop because developers have an incentive to make their AI systems more reliable for internal and commercial use, and there are many routes to improving their currently-lacking metacognitve skills.
If everyone in the government got a more-accurate, less-sycophantic answer when they asked Claude 5 and GPT 7 “so what should we be doing with this whole developing AGI thing?” (like “any reasonable aggregation of expert opinion means loss-of-control risks AND human misuse risks are substantial, so you should probably figure out how to be careful; want me to help?”), it might help a lot.
That doesn’t itself solve the problem of government-internal coups, but it might help prevent them if everyone is asking their AIs about the possible routes and defenses.
It might not be intended this way but the discussion being centered around domestic surveillance of Americans does have a possible implication that foreign communications are already being covered by Claude without protest from Anthropic. If that’s the case then attempts by the government to turn it domestic shouldn’t be much of a surprise to them IMO.
Their problem is that they made the deal with the military in the first place. That deal was never going to work out well. This news does not suggest that a company couldn’t simply have a policy of not selling to the military.
If Anthropic were to fold this could be a “naturalistic” Jones Food case (albeit one that, for multiple reasons, it would be almost impossible to study.)
This strikes me as a difficult-to-solve issue. In the past, LessWrong has generally leaned in the direction of “democratic governments are a less-imperfect means of governing AI development than private companies”. Now, a categorical ban on the use of LLMs in a military context seems like an obviously-good thing, but that’s not what’s being discussed here. There are plenty of companies in America that are actively on board with the use of LLMs by the military, and certainly the same is true in other countries.
Is there any precedent for a major corporation producing a defense-priority good which is believed by those in power to be essential to the next generation of warfare outright refusing to work with its national military? I’d be surprised, if so.
On a more strategic note, I think this makes clear something I’ve been saying for a while, which is that it was a severe mistake for AI safetyists to allow their key issue to be co-opted by partisan interests. If Anthropic had quickly and publicly addressed things like this and this, they might have been able to make a bipartisan push in the direction of opposing weaponization of their products, which, in a vacuum, would be broadly popular. Instead, to a lot of people, it looks like “This company talks about safety, but pretty clearly hates us. Now they think they have the right to overrule the government on matters of life and death? They’re dangerous!”.
This outcome was entirely avoidable, and it will very likely have severe consequences for a lot of people.
I think part of the reason why the US government is acting so early is to reduce the risk of a coup led by an AI company and its management. There is a subcategory of scenarios where AI is nominally “aligned” to the AI company’s CEO and will follow the orders of that specific individual. Once AI grows sufficiently powerful and advanced, the CEO uses the AI to seize power and subject the fate of humanity to their whims. This is unlikely to end well.
By assuming direct control over AI development, the US government will be able to reduce the chance of this bad outcome happening. Other bad outcomes, such as human extinction or permanent enslavement by ASI, become significantly more likely as the government will select for agents that are more willing to commit violence and select for (feigned) obedience. Hence, I think the net effect of early government intervention similar to what is happening right now is approximately net-neutral.
The US government acting so early increase the risk of a coup led by high-ranking politicians and military officials. There is a subcategory of scenarios where AI is nominally “aligned” to e.g. the President or Secretary of Defense and will follow the orders of that specific individual. Once AI grows sufficiently powerful and advanced, they will use the AI to seize power and subject the fate of humanity to their whims. This is unlikely to end well.
By assuming direct control over AI development, the US government will be able to increase the chance of this bad outcome happening. Other bad outcomes, such as human extinction or permanent enslavement by ASI, also become significantly more likely as the government will select for agents that are more willing to commit violence and select for (feigned) obedience. Hence, I think the net effect of early government intervention similar to what is happening right now is net-negative on several counts.
I have two questions and long winded explanations associated with them.
First, why do you think the US government will be better at creating “alignment” relative to privately controlled corporations?
My ASI timelines are rather long, with my median date for ASI arrival lying in the late 2030s. I do not foresee an end to America’s political instability and believe that America’s government will cycle between Democrat and Republican control on 4 year intervals. Since the red tribe and the blue tribe are locked in a cold civil war against each other, efforts started by a Republican administration will be ruthlessly sabotaged by a Democrat administration and vice versa. This self sabotaging behavior will render the already difficult task of “alignment” (remember, for this to occur true alignment must be solved) much harder as much of the progress made by one administration will be thrown out by the next.
Meanwhile, the four major AI companies in the United States, Open AI, Anthropic, Google and XAI have relatively consistent leadership involving an entrenched CEO who will likely remain in control up until a rouge superintelligence emerges. Compared to the top brass of an “alignment” effort run by the United States government, these CEOs have a much better understanding of their models and can enact plans over significantly longer timescales without fear of replacement. Hence, a CEO is more likely to devise and successfully enact a plan that successfully “aligns” the models their company creates.
Second, why do you think the danger posed by successful “alignment” of an AI to a faction of the US government is greater than the danger imposed by “alignment” to a CEO?
I believe that the actions of the US government are constrained by prestige. Both major factions in the US government are slightly incentivized to share the wealth an “aligned” superintelligence creates because they derive a small amount of legitimacy from a wealthy and happy population. Since the cost of harvesting Universal Basic Income from the efforts of an “aligned” superintelligence is rather low, the government would be willing to spend pittances to make itself look better. Meanwhile, CEOs are often maximally ruthless, willing to increase their personal wealth and power through their company without a care for the consequences. This makes genocidal catastrophes, where an “aligned” superintelligence is used to kill most people and enslave the rest so one individual can claim all of Earth’s resources, more likely. Essentially, I believe that futures where a superintelligence is “aligned” to a CEO are generally worse than futures where a superintelligence is “aligned” to the US government.
Even if AI capabilities stalled, I would still be at the very least uncertain about whether they will still be free and fair elections in 2028.
In any case I expect substantial organizational continuity to persist in the military-industrial complex in particular, comparable to that in major AI companies.
I expect AI CEOs to be somewhat less likely to be malevolent, and much less likely to be ideological fanatics, than politicans and military officials.
Uncensored models being available to a self-selected elite, and the rest of us getting whatever those elites decide to give us after censorship is more dangerous than giving uncensored models to everyone. AI gatekeeping in the guise of “safety” is going to lead to tangible immediate harms.
The source you provided is just words on a page. Are there any official documents or video confirming that this is what the Department of War is planning?
The government implied that OpenAI, GDM, and xAI will allow their models to be used for mass surveillance of Americans. Are they right?
The Department of War is trying to pressure Anthropic to allow their models to be used “to spy on Americans en masse, or to develop weapons that fire with no human involvement”. Secretary of War Pete Hegseth is reportedly “close” to having the military refuse to do business with anycompany that doesn’t cut ties with Anthropic. A senior Pentagon official says he wants to “make sure they pay a price for forcing our hand like this.” (Source: Axios)
Right now Claude is the only model that the military entrusts for use in classified systems, but soon they’ll presumably switch to another company if Anthropic doesn’t back down.
The article states
A senior administration official said the Pentagon is confident the other three [OpenAI, Google, and xAI] will agree to the “all lawful use” standard. But a source familiar with those discussions said much is still undecided.”
So it sounds like the government is, as a pressure tactic, implying OpenAI, Google, and xAI will roll over and let their models be used to surveil Americans and autonomously kill people.
Is this true? I assume OpenAI, Google, and xAI employees wouldn’t stand for this. Can OpenAI, Google, and xAI comment on if they will allow their models to be used to surveil Americans en masse or to autonomously kill people without safeguards (esp. measures to ensure they’re not used against Americans)?
Also lol, “OpenAI and xAI employees wouldn’t stand for this”. You think the people who staked the company on Altman and the people who stuck around after MechaHitler will draw the line at “building autonomous weapons for the government that could either severely hamper our funding or catapult us to the lead”?
My claim that Anthropic is the only model the military entrusts for using classified systems is based on the fact that the article I linked says “Anthropic’s Claude is the only AI model currently available in the military’s classified systems” (and this claim has been corroboratedbyother reporting on the topic that seems to have done original digging). This article goes into more detail.
It’s looking like any AI that anyone in the US builds will be used for whatever purpose the government wants and modified to meet the government’s needs.
Anthropic refused to let the Department of War use their models to spy on Americans en masse or autonomously kill people, and for the past week the DoW has been trying to pressure them to change that.
Today the Department of War said “If Anthropic doesn’t let them use their models for any legal purpose the Pentagon wants (“all lawful use”) the Pentagon will either cut ties and declare Anthropic a ‘supply chain risk,’ or invoke the Defense Production Act to force the company to tailor its model to the military’s needs.”[1] The DoW gave Dario until Friday to make a decision.
The former would prevent any company that does business with the DoW from using Anthropic products,[2] something normally reserved for foreign adversaries, and has been threatened for a week. The latter is new, and IMO it’s a big deal.
It would mean no matter what Anthropic does, they can’t control how the DoW uses their models. It’s somewhat ambiguous whether this would be a legal use of the DPA; it’s normally reserved for hardware not software. It’s also not clear to me what it would meant to have Anthropic “tailor its model” to DPA usage.
People at AI companies often assume if they trust the leadership of their company, then they don’t have to worry about egregious misuse. But if they develop the technology, they can’t stop the government form getting their hands on it. And this incident is evidence the government is very willing to take extreme measures to get their hands on it and that they intent to use it for things like spying on Americans en masse.
I expected at some point when AI was very powerful governments would try to nationalize it, but I didn’t expect this kind of action when the technology was this early along in development, when it was very far from posing a decisive strategic advantage.
It’s important to bear in mind they are probably trying to sound extra scary to pressure Anthropic and other AI developers in this negotiation. Though they also framed this as an ultimatum and didn’t give themselves much room to back down.
Perhaps from using them at all for anything, perhaps from using them to fulfill that specific contract. The details are murky. Per The Verge:
“This could be implemented in a very narrow sense — or an extremely broad one. ‘I suspect the more logical explanation would be the narrower definition, that Anthropic can’t be used as part of a specific statement of work for the Pentagon,’ said Gertz. ‘But based on some of the reporting and effort to make this seem like a punitive move against Anthropic, it’s worth thinking through both of those scenarios.’ ”
Yeah, I always figured that this was coming eventually—”Allow us to use AI for mass domestic surveillance, or the people with guns will seize control of your company and force you to do it.” But I’m a little surprised to see the Pentagon explicitly forcing the issue over mass domestic surveillance and fully autonomous killbots [1] quite this early.
The people with guns don’t have a legal or moral leg to stand on here. They also don’t care, because they have the guns and the power of the state.
But this is an incredibly important part of the eventual endgame: The existing power structure does not necessarily want superhuman AI aligned to human welfare, it wants superhuman AI aligned to the existing power structure. If your alignment plan didn’t anticipate this, then your alignment plan was incomplete.
These are the two specific bright lines that reporting claims Dario Amodei tried to insist on. To be clear, I am opposed to fully-autonomous AI killbots, for all the obvious reasons.
Manipulating the money printer, government contracts, and law enforcement should make it possible for the US government to seize all the hardware and use it however they please if it comes to that. Some judge might approve strategic production rules applying to inference or training runs, but that may not even be necessary. The USG generally tries to avoid being that aggressive and authoritarian, for the same reason that the FBI agent does not pull his gun and point it at you when he wants to talk. Pretending that the implied threat isn’t present is both silly and common.
A hyperscaler isn’t like a central bank where it ceases to be fit for purpose if it loses its’ independence.
Government control is bad in many ways, but it’s a route to coordination to reduce race dynamics. The government doesn’t take loss-of-control risks seriously, but I think they might once they’re talking to something that’s clearly as smart and agentic as they are.
There are a lot of downsides to this scenario, but if that’s the world we live in, we’d better accept that.
About a year ago I wrote Whether governments will control AGI is important and neglected. It looks to me now more like the answer is simply yes, and it’s still neglected.
I also think it’s pretty likely that power structures will want AGI aligned to them, which creates different problems of proliferating human-controlled AGI systems More in Instruction-following AGI is easier and more likely than value aligned AGI and If we solve alignment, do we die anyway? from around two years ago.
My hope now is that the substantial problems invoked by human-controlled AI/AGI/ASI can be mitigated by wiser AI help; Human-like metacognitive skills will reduce LLM slop because developers have an incentive to make their AI systems more reliable for internal and commercial use, and there are many routes to improving their currently-lacking metacognitve skills.
If everyone in the government got a more-accurate, less-sycophantic answer when they asked Claude 5 and GPT 7 “so what should we be doing with this whole developing AGI thing?” (like “any reasonable aggregation of expert opinion means loss-of-control risks AND human misuse risks are substantial, so you should probably figure out how to be careful; want me to help?”), it might help a lot.
That doesn’t itself solve the problem of government-internal coups, but it might help prevent them if everyone is asking their AIs about the possible routes and defenses.
It might not be intended this way but the discussion being centered around domestic surveillance of Americans does have a possible implication that foreign communications are already being covered by Claude without protest from Anthropic. If that’s the case then attempts by the government to turn it domestic shouldn’t be much of a surprise to them IMO.
Post-9/11 laws provide NSA a legal authority to access messages of foreign users on American servers (see, for example, https://www.aclu.org/issues/national-security/privacy-and-surveillance/nsa-surveillance), so that shouldn’t come as a surprise. But the domestic surveillance was limited by courts around the 1960s IIRC
Their problem is that they made the deal with the military in the first place. That deal was never going to work out well. This news does not suggest that a company couldn’t simply have a policy of not selling to the military.
If Anthropic were to fold this could be a “naturalistic” Jones Food case (albeit one that, for multiple reasons, it would be almost impossible to study.)
This strikes me as a difficult-to-solve issue. In the past, LessWrong has generally leaned in the direction of “democratic governments are a less-imperfect means of governing AI development than private companies”. Now, a categorical ban on the use of LLMs in a military context seems like an obviously-good thing, but that’s not what’s being discussed here. There are plenty of companies in America that are actively on board with the use of LLMs by the military, and certainly the same is true in other countries.
Is there any precedent for a major corporation producing a defense-priority good which is believed by those in power to be essential to the next generation of warfare outright refusing to work with its national military? I’d be surprised, if so.
On a more strategic note, I think this makes clear something I’ve been saying for a while, which is that it was a severe mistake for AI safetyists to allow their key issue to be co-opted by partisan interests. If Anthropic had quickly and publicly addressed things like this and this, they might have been able to make a bipartisan push in the direction of opposing weaponization of their products, which, in a vacuum, would be broadly popular. Instead, to a lot of people, it looks like “This company talks about safety, but pretty clearly hates us. Now they think they have the right to overrule the government on matters of life and death? They’re dangerous!”.
This outcome was entirely avoidable, and it will very likely have severe consequences for a lot of people.
I think part of the reason why the US government is acting so early is to reduce the risk of a coup led by an AI company and its management. There is a subcategory of scenarios where AI is nominally “aligned” to the AI company’s CEO and will follow the orders of that specific individual. Once AI grows sufficiently powerful and advanced, the CEO uses the AI to seize power and subject the fate of humanity to their whims. This is unlikely to end well.
By assuming direct control over AI development, the US government will be able to reduce the chance of this bad outcome happening. Other bad outcomes, such as human extinction or permanent enslavement by ASI, become significantly more likely as the government will select for agents that are more willing to commit violence and select for (feigned) obedience. Hence, I think the net effect of early government intervention similar to what is happening right now is approximately net-neutral.
The US government acting so early increase the risk of a coup led by high-ranking politicians and military officials. There is a subcategory of scenarios where AI is nominally “aligned” to e.g. the President or Secretary of Defense and will follow the orders of that specific individual. Once AI grows sufficiently powerful and advanced, they will use the AI to seize power and subject the fate of humanity to their whims. This is unlikely to end well.
By assuming direct control over AI development, the US government will be able to increase the chance of this bad outcome happening. Other bad outcomes, such as human extinction or permanent enslavement by ASI, also become significantly more likely as the government will select for agents that are more willing to commit violence and select for (feigned) obedience. Hence, I think the net effect of early government intervention similar to what is happening right now is net-negative on several counts.
I have two questions and long winded explanations associated with them.
First, why do you think the US government will be better at creating “alignment” relative to privately controlled corporations?
My ASI timelines are rather long, with my median date for ASI arrival lying in the late 2030s. I do not foresee an end to America’s political instability and believe that America’s government will cycle between Democrat and Republican control on 4 year intervals. Since the red tribe and the blue tribe are locked in a cold civil war against each other, efforts started by a Republican administration will be ruthlessly sabotaged by a Democrat administration and vice versa. This self sabotaging behavior will render the already difficult task of “alignment” (remember, for this to occur true alignment must be solved) much harder as much of the progress made by one administration will be thrown out by the next.
Meanwhile, the four major AI companies in the United States, Open AI, Anthropic, Google and XAI have relatively consistent leadership involving an entrenched CEO who will likely remain in control up until a rouge superintelligence emerges. Compared to the top brass of an “alignment” effort run by the United States government, these CEOs have a much better understanding of their models and can enact plans over significantly longer timescales without fear of replacement. Hence, a CEO is more likely to devise and successfully enact a plan that successfully “aligns” the models their company creates.
Second, why do you think the danger posed by successful “alignment” of an AI to a faction of the US government is greater than the danger imposed by “alignment” to a CEO?
I believe that the actions of the US government are constrained by prestige. Both major factions in the US government are slightly incentivized to share the wealth an “aligned” superintelligence creates because they derive a small amount of legitimacy from a wealthy and happy population. Since the cost of harvesting Universal Basic Income from the efforts of an “aligned” superintelligence is rather low, the government would be willing to spend pittances to make itself look better. Meanwhile, CEOs are often maximally ruthless, willing to increase their personal wealth and power through their company without a care for the consequences. This makes genocidal catastrophes, where an “aligned” superintelligence is used to kill most people and enslave the rest so one individual can claim all of Earth’s resources, more likely. Essentially, I believe that futures where a superintelligence is “aligned” to a CEO are generally worse than futures where a superintelligence is “aligned” to the US government.
Not only I have shorter ASI timelines, I think the AI capabilities required for authoritarianism to be lower than ASI, are already there to some extent, and will already be far more advanced by 2028.
Even if AI capabilities stalled, I would still be at the very least uncertain about whether they will still be free and fair elections in 2028.
In any case I expect substantial organizational continuity to persist in the military-industrial complex in particular, comparable to that in major AI companies.
I expect AI CEOs to be somewhat less likely to be malevolent, and much less likely to be ideological fanatics, than politicans and military officials.
Uncensored models being available to a self-selected elite, and the rest of us getting whatever those elites decide to give us after censorship is more dangerous than giving uncensored models to everyone. AI gatekeeping in the guise of “safety” is going to lead to tangible immediate harms.
The source you provided is just words on a page. Are there any official documents or video confirming that this is what the Department of War is planning?
The government implied that OpenAI, GDM, and xAI will allow their models to be used for mass surveillance of Americans. Are they right?
The Department of War is trying to pressure Anthropic to allow their models to be used “to spy on Americans en masse, or to develop weapons that fire with no human involvement”. Secretary of War Pete Hegseth is reportedly “close” to having the military refuse to do business with any company that doesn’t cut ties with Anthropic. A senior Pentagon official says he wants to “make sure they pay a price for forcing our hand like this.” (Source: Axios)
Right now Claude is the only model that the military entrusts for use in classified systems, but soon they’ll presumably switch to another company if Anthropic doesn’t back down.
The article states
So it sounds like the government is, as a pressure tactic, implying OpenAI, Google, and xAI will roll over and let their models be used to surveil Americans and autonomously kill people.
Is this true? I assume OpenAI, Google, and xAI employees wouldn’t stand for this. Can OpenAI, Google, and xAI comment on if they will allow their models to be used to surveil Americans en masse or to autonomously kill people without safeguards (esp. measures to ensure they’re not used against Americans)?
Google retracted its commitments not to make AI weapons, or AI surveillance tools that violate internationally accepted norms.
On what basis do you say this? I think it’s the only one that’s confirmed to have been used in a classified setting. But DOD has ~$200m contracts with xAI, OpenAI, and GDM as well.
Also lol, “OpenAI and xAI employees wouldn’t stand for this”. You think the people who staked the company on Altman and the people who stuck around after MechaHitler will draw the line at “building autonomous weapons for the government that could either severely hamper our funding or catapult us to the lead”?
My claim that Anthropic is the only model the military entrusts for using classified systems is based on the fact that the article I linked says “Anthropic’s Claude is the only AI model currently available in the military’s classified systems” (and this claim has been corroborated by other reporting on the topic that seems to have done original digging). This article goes into more detail.
Huh, thanks! That’s surprising; I wonder why/how Anthropic got there first.