Google didn’t necessarily even break a commitment? The commitment mentioned in the article is to “publicly report model or system capabilities.” That doesn’t say it has to be done at the time of public deployment.
The White House voluntary commitments included a commitment to “publish reports for all new significant model public releases”; same deal there.
Possibly Google broke a different commitment (mentioned in the open letter): “Assess the risks posed by their frontier models or systems across the AI lifecycle, including before deploying that model or system.” Depends on your reading of “assess the risks” plus facts which I don’t recall off the top of my head.
Other companies are doing far worse in this dimension. At worst Google is 3rd-best in publishing eval results. Meta and xAI are far worse.
2. Google didn’t necessarily even break a commitment? The commitment mentioned in the article is to “publicly report model or system capabilities.” That doesn’t say it has to be done at the time of public deployment.
This document linked on the open letter page gives a precise breakdown of exactly what the commitments were and how Google broke them (both in spirit and by the letter).[1] The summary is this:
Google violated the spirit of commitment I by publishing its first safety report almost a month after public availability and not mentioning external testing in their initial report.
Google explicitly violated commitment VIII by not stating whether governments are involved in safety testing, even after being asked directly by reporters.
But in fact the letter actually understates the degree to which Google DeepMind violated the commitments. The real story from this article is that GDM confirmed to Time that they didn’t provide any pre-deployment access to UK AISI:
However, Google says it only shared the model with the U.K. AI Security Institute after Gemini 2.5 Pro was released on March 25.
If UK AISI doesn’t have pre-deployment access, a large portion of their whole raison d’être is nullified.
Assess the risks posed by their frontier models or systems across the AI lifecycle, including before deploying that model or system… They should also consider results from internal and external evaluations as appropriate, such as by independent third-party evaluators, their home governments, and other bodies their governments deem appropriate.
And if they didn’t give pre-deployment access to UK AISI, it’s a fairly safe bet they didn’t provide pre-deployment access to any other external evaluator.
The violation is also explained, although less clearly, in the Time article:
The update also stated the use of “third-party external testers,” but did not disclose which ones or whether the U.K. AI Security Institute had been among them—which the letter also cites as a violation of Google’s pledge.
After previously failing to address a media request for comment on whether it had shared Gemini 2.5 Pro with governments for safety testing...
Thanks. Sorry for criticizing without reading everything. I agree that, like, on balance, GDM didn’t fully comply with the Seoul commitments re Gemini 2.5. Maybe I just don’t care much about these particular commitments.
3. Other companies are doing far worse in this dimension. At worst Google is 3rd-best in publishing eval results. Meta and xAI are far worse.
Some reasons for focusing Google DeepMind in particular:
The letter was organized by PauseAI UK and signed by UK politicians. GDM is the only frontier AI company headquartered in the UK.
Meta and xAI already have a bad reputation for their safety practices, while GDM had a comparatively good reputation and most people were unaware of their violation of the Frontier AI Safety Commitments.
[Writing off based on the quick take, haven’t looked into the linked thing.]
Google didn’t necessarily even break a commitment? The commitment mentioned in the article is to “publicly report model or system capabilities.” That doesn’t say it has to be done at the time of public deployment.
I think this statement lends itself to being by default interpreted as “inform the public about model/system capabilities in time” (because if they don’t do it “in time”, then what’s the point?), and the most “natural” “in-time” time is the time of deployment?
I announce that I’m committing to X. I can expect that most people will understand this to mean “I commit to Y” where X→Y is a natural (~unconscious?) inference for a human to make. And then I don’t do Y and defend myself by saying, “The only thing I committed to was X, why all the fuss about me not Y-ing?”.
Other companies are doing far worse in this dimension. At worst Google is 3rd-best in publishing eval results. Meta and xAI are far worse.
There might be a contextualizer-y justification for picking on Google more because they are more ahead than Meta and xAI, AI-wise.
I agree. If Google wanted to join the commitments but not necessarily publish eval results by the time of external deployment, it should have clarified “we’ll publish within 2 months after external deployment” or “we’ll do evals on our most powerful model at least every 4 months rather than doing one round of evals per model” or something.
I’m not sure why this would make you not feel good about the critique or implicit ask of the letter. Sure, maybe internal deployment transparency would be better, but public deployment transparency is better than nothing.
And that’s where the leverage is right now. Google made a commitment to transparency about external deployments, not internal deployments. And they should be held to that commitment or else we establish the precedent that AI safety commitments don’t matter and can be ignored.
I think I’d prefer “within a month after external deployment” over “by the time of external deployment” because I expect the latter will lead to (1) evals being rushed and (2) safety people being forced to prioritize poorly.
To clarify, the primary complaint from my perspective is not that they published the report a month after external deployment per se, but that the timing of the report indicates that they did not perform thorough pre-deployment testing (and zero external testing).
And the focus on pre-deployment testing is not really due to any opinion about the relative benefits of pre- vs. post- deployment testing, but because they committed to doing pre-deployment testing, so it’s important that they in fact do pre-deployment testing.
Some of my friends are signal-boosting this new article: 60 U.K. Lawmakers Accuse Google of Breaking AI Safety Pledge. See also the open letter. I don’t feel good about this critique or the implicit ask.
Sharing information on capabilities is good but public deployment is a bad time for that, in part because most risk comes from internal deployment.
Google didn’t necessarily even break a commitment? The commitment mentioned in the article is to “publicly report model or system capabilities.” That doesn’t say it has to be done at the time of public deployment.
The White House voluntary commitments included a commitment to “publish reports for all new significant model public releases”; same deal there.
Possibly Google broke a different commitment (mentioned in the open letter): “Assess the risks posed by their frontier models or systems across the AI lifecycle, including before deploying that model or system.” Depends on your reading of “assess the risks” plus facts which I don’t recall off the top of my head.
Other companies are doing far worse in this dimension. At worst Google is 3rd-best in publishing eval results. Meta and xAI are far worse.
This document linked on the open letter page gives a precise breakdown of exactly what the commitments were and how Google broke them (both in spirit and by the letter).[1] The summary is this:
But in fact the letter actually understates the degree to which Google DeepMind violated the commitments. The real story from this article is that GDM confirmed to Time that they didn’t provide any pre-deployment access to UK AISI:
If UK AISI doesn’t have pre-deployment access, a large portion of their whole raison d’être is nullified.
Google withholding access is quite strongly violating the spirit of commitment I of the Frontier AI Safety Commitments:
And if they didn’t give pre-deployment access to UK AISI, it’s a fairly safe bet they didn’t provide pre-deployment access to any other external evaluator.
The violation is also explained, although less clearly, in the Time article:
Thanks. Sorry for criticizing without reading everything. I agree that, like, on balance, GDM didn’t fully comply with the Seoul commitments re Gemini 2.5. Maybe I just don’t care much about these particular commitments.
Some reasons for focusing Google DeepMind in particular:
The letter was organized by PauseAI UK and signed by UK politicians. GDM is the only frontier AI company headquartered in the UK.
Meta and xAI already have a bad reputation for their safety practices, while GDM had a comparatively good reputation and most people were unaware of their violation of the Frontier AI Safety Commitments.
[Writing off based on the quick take, haven’t looked into the linked thing.]
I think this statement lends itself to being by default interpreted as “inform the public about model/system capabilities in time” (because if they don’t do it “in time”, then what’s the point?), and the most “natural” “in-time” time is the time of deployment?
I announce that I’m committing to X. I can expect that most people will understand this to mean “I commit to Y” where X→Y is a natural (~unconscious?) inference for a human to make. And then I don’t do Y and defend myself by saying, “The only thing I committed to was X, why all the fuss about me not Y-ing?”.
There might be a contextualizer-y justification for picking on Google more because they are more ahead than Meta and xAI, AI-wise.
I agree. If Google wanted to join the commitments but not necessarily publish eval results by the time of external deployment, it should have clarified “we’ll publish within 2 months after external deployment” or “we’ll do evals on our most powerful model at least every 4 months rather than doing one round of evals per model” or something.
I’m not sure why this would make you not feel good about the critique or implicit ask of the letter. Sure, maybe internal deployment transparency would be better, but public deployment transparency is better than nothing.
And that’s where the leverage is right now. Google made a commitment to transparency about external deployments, not internal deployments. And they should be held to that commitment or else we establish the precedent that AI safety commitments don’t matter and can be ignored.
I think I’d prefer “within a month after external deployment” over “by the time of external deployment” because I expect the latter will lead to (1) evals being rushed and (2) safety people being forced to prioritize poorly.
To clarify, the primary complaint from my perspective is not that they published the report a month after external deployment per se, but that the timing of the report indicates that they did not perform thorough pre-deployment testing (and zero external testing).
And the focus on pre-deployment testing is not really due to any opinion about the relative benefits of pre- vs. post- deployment testing, but because they committed to doing pre-deployment testing, so it’s important that they in fact do pre-deployment testing.