I’m not saying we should optimize for convincing every NYT reader (nor do I think I suggested this?). I’m saying that we should optimize for convincing the general public that these risks should be taken seriously. Due to the public’s (reasonable!) priors about anthropic’s announcements being self-serving, the standard of evidence to convince a by-default-skeptical person is higher.
You talk about multiple different actors. One the one hand, there might be the LessWrong community and on the other hand there’s Anthropic.
When Anthropic makes those decisions, the general public isn’t the key audience. Various people in the security community are a key audience. The government is a key audience. Maybe, companies who want to switch to AI agents and investors are also a key audience.
I hear the point you’re making. But I guess it’s not clear to me who the key audience of the model cards is? The red-teaming demonstrations in them (e.g., blackmailing) are exceptionally valuable for the safety community and as ways to engage the general public. It’s not obvious to me that these are directed at the government, investors or companies who want to switch to AI agents?
From the safety community it would be a very strange thing to say something like “Companies should only tell us about safety problems they are having when they can provide extraordinary proof that they have safety problems.”
That’s not how you handle companies dealing with product safety in any domain. You want companies to share information about safety issues even if the evidence for those safety issues is not overwhelming.
I’m not saying we should optimize for convincing every NYT reader (nor do I think I suggested this?). I’m saying that we should optimize for convincing the general public that these risks should be taken seriously. Due to the public’s (reasonable!) priors about anthropic’s announcements being self-serving, the standard of evidence to convince a by-default-skeptical person is higher.
You talk about multiple different actors. One the one hand, there might be the LessWrong community and on the other hand there’s Anthropic.
When Anthropic makes those decisions, the general public isn’t the key audience. Various people in the security community are a key audience. The government is a key audience. Maybe, companies who want to switch to AI agents and investors are also a key audience.
I hear the point you’re making. But I guess it’s not clear to me who the key audience of the model cards is? The red-teaming demonstrations in them (e.g., blackmailing) are exceptionally valuable for the safety community and as ways to engage the general public. It’s not obvious to me that these are directed at the government, investors or companies who want to switch to AI agents?
From the safety community it would be a very strange thing to say something like “Companies should only tell us about safety problems they are having when they can provide extraordinary proof that they have safety problems.”
That’s not how you handle companies dealing with product safety in any domain. You want companies to share information about safety issues even if the evidence for those safety issues is not overwhelming.