Imo the a natural reading of “public deployment” in the Anthropic RSP is about whether a member of the general public can access the model, which isn’t the case currently.
I agree that the literal text of Section 1.2.2 weakly implies that this is considered a public deployment, but it seems way more likely that there just wasn’t that much thought put into that one particular paragraph, e.g. maybe it was copied from an earlier system card and nobody noticed the implication, maybe it was drafted earlier when they planned to release to the public and it didn’t get changed when the plans changed, etc.
Anthropic really needs to cut it out with the pattern of
sort of imply they will do X
people call them out on not doing X
Anthropic* retreats to “technically we never said we would do X!!”
At this point I have very little patience for any claim that people misunderstood Anthropic’s commitment and actually it was weaker than that. If Anthropic doesn’t want people holding them to commitments they didn’t intend to make, then they need to stop making ambiguous commitments.
*I realize you [Rohin] don’t work at Anthropic, my point is that I don’t think Anthropic is allowed to use the defense that you’re suggesting they could use.
I think the strongest argument here is that Anthropic themselves refer to the section of the RSP that says they have to do a risk report when they “publicly deploy” a model, when they talk about why they are releasing the current risk report:
And if we release a model that is “significantly more capable” than those discussed in the prior Risk Report, we must “publish a discussion (in our System Card or elsewhere) of how that model’s capabilities and propensities affect or change analysis in the Risk Report.”
“significantly more capable” is a quote from this paragraph:
When we publicly deploy a model that we determine is significantly more capable than any of the models covered in the most recent Risk Report, we will publish a discussion (in our System Card or elsewhere) of how that model’s capabilities and propensities affect or change analysis in the Risk Report.
It’s not like a perfectly airtight case, but it seems to me that Anthropic is saying in the first paragraph that they are considering the Mythos release to be the kind of thing that would trigger the second paragraph, which would be a “public deployment”.
I agree the common-sense reading of “public deployment” could reasonably not apply to the present situation (though it’s IMO a bit of a stretch), but I think given these paragraphs, it seems like Anthropic themselves think it met the relevant threshold.
I agree it’s a reasonable reading, but I’m not sure I’m sold on “natural”, since in that case there’s no category defined for the situation Anthropic finds themselves in with Mythos.
It seems a little awkard for Anthropic to have deployed the model to a small number of external parties and for the RSP to have nothing to say about that, while it does have things to say about “internal only” and “generally available” deployments. (Though I do think this kind of categorization should be driven by risk modeling, not vice-versa.)
Separately, I think both the text of Section 1.2.2 and the entire Alignment Risk Update being conducted suggest that Anthropic was treating something about this release the way they’d treat a public deployment, but as you say it’s possible they were planning on a general release but changed their minds, and so happened to have done all the necessary work already.
Imo
thea natural reading of “public deployment” in the Anthropic RSP is about whether a member of the general public can access the model, which isn’t the case currently.I agree that the literal text of Section 1.2.2 weakly implies that this is considered a public deployment, but it seems way more likely that there just wasn’t that much thought put into that one particular paragraph, e.g. maybe it was copied from an earlier system card and nobody noticed the implication, maybe it was drafted earlier when they planned to release to the public and it didn’t get changed when the plans changed, etc.
Anthropic really needs to cut it out with the pattern of
sort of imply they will do X
people call them out on not doing X
Anthropic* retreats to “technically we never said we would do X!!”
At this point I have very little patience for any claim that people misunderstood Anthropic’s commitment and actually it was weaker than that. If Anthropic doesn’t want people holding them to commitments they didn’t intend to make, then they need to stop making ambiguous commitments.
*I realize you [Rohin] don’t work at Anthropic, my point is that I don’t think Anthropic is allowed to use the defense that you’re suggesting they could use.
I think the strongest argument here is that Anthropic themselves refer to the section of the RSP that says they have to do a risk report when they “publicly deploy” a model, when they talk about why they are releasing the current risk report:
“significantly more capable” is a quote from this paragraph:
It’s not like a perfectly airtight case, but it seems to me that Anthropic is saying in the first paragraph that they are considering the Mythos release to be the kind of thing that would trigger the second paragraph, which would be a “public deployment”.
I agree the common-sense reading of “public deployment” could reasonably not apply to the present situation (though it’s IMO a bit of a stretch), but I think given these paragraphs, it seems like Anthropic themselves think it met the relevant threshold.
I agree it’s a reasonable reading, but I’m not sure I’m sold on “natural”, since in that case there’s no category defined for the situation Anthropic finds themselves in with Mythos.
It seems a little awkard for Anthropic to have deployed the model to a small number of external parties and for the RSP to have nothing to say about that, while it does have things to say about “internal only” and “generally available” deployments. (Though I do think this kind of categorization should be driven by risk modeling, not vice-versa.)
Separately, I think both the text of Section 1.2.2 and the entire Alignment Risk Update being conducted suggest that Anthropic was treating something about this release the way they’d treat a public deployment, but as you say it’s possible they were planning on a general release but changed their minds, and so happened to have done all the necessary work already.