Data from Firefoxsuggests that Mythos found 10x the amount of security vulnerabilities as the previous version Opus 4.6.
We do know that the Chinese hackers previously used Anthropics models to successfully hack targets. I think it’s plausible that the Mythos could have given various people who are willing to hack a bunch of “zero days” who likely won’t stay zero days very long and are thus cheap to burn along with an easy way to create exploits for them.
One aspect of project Glassdoor seems to be that Anthropic didn’t just have Mythos as a model but also spent 100$ million in credits for cybersecurity and another 4$ million in donations to Open Source projects to process the found claims.
No, that’s not how it works in safety regulation. Generally, the precautionary principle suggests that we should not require extraordinary evidence before taking steps to prevent risks. The serious thing to do is to take actions for safeties sake even if you don’t have the extraordinary evidence to convince everyone.
From Anthropics perspective, optimizing for convincing every NYT reader is completely unnecessary.
I’m not saying we should optimize for convincing every NYT reader (nor do I think I suggested this?). I’m saying that we should optimize for convincing the general public that these risks should be taken seriously. Due to the public’s (reasonable!) priors about anthropic’s announcements being self-serving, the standard of evidence to convince a by-default-skeptical person is higher.
You talk about multiple different actors. One the one hand, there might be the LessWrong community and on the other hand there’s Anthropic.
When Anthropic makes those decisions, the general public isn’t the key audience. Various people in the security community are a key audience. The government is a key audience. Maybe, companies who want to switch to AI agents and investors are also a key audience.
I hear the point you’re making. But I guess it’s not clear to me who the key audience of the model cards is? The red-teaming demonstrations in them (e.g., blackmailing) are exceptionally valuable for the safety community and as ways to engage the general public. It’s not obvious to me that these are directed at the government, investors or companies who want to switch to AI agents?
From the safety community it would be a very strange thing to say something like “Companies should only tell us about safety problems they are having when they can provide extraordinary proof that they have safety problems.”
That’s not how you handle companies dealing with product safety in any domain. You want companies to share information about safety issues even if the evidence for those safety issues is not overwhelming.
Data from Firefox suggests that Mythos found 10x the amount of security vulnerabilities as the previous version Opus 4.6.
We do know that the Chinese hackers previously used Anthropics models to successfully hack targets. I think it’s plausible that the Mythos could have given various people who are willing to hack a bunch of “zero days” who likely won’t stay zero days very long and are thus cheap to burn along with an easy way to create exploits for them.
One aspect of project Glassdoor seems to be that Anthropic didn’t just have Mythos as a model but also spent 100$ million in credits for cybersecurity and another 4$ million in donations to Open Source projects to process the found claims.
No, that’s not how it works in safety regulation. Generally, the precautionary principle suggests that we should not require extraordinary evidence before taking steps to prevent risks. The serious thing to do is to take actions for safeties sake even if you don’t have the extraordinary evidence to convince everyone.
From Anthropics perspective, optimizing for convincing every NYT reader is completely unnecessary.
I’m not saying we should optimize for convincing every NYT reader (nor do I think I suggested this?). I’m saying that we should optimize for convincing the general public that these risks should be taken seriously. Due to the public’s (reasonable!) priors about anthropic’s announcements being self-serving, the standard of evidence to convince a by-default-skeptical person is higher.
You talk about multiple different actors. One the one hand, there might be the LessWrong community and on the other hand there’s Anthropic.
When Anthropic makes those decisions, the general public isn’t the key audience. Various people in the security community are a key audience. The government is a key audience. Maybe, companies who want to switch to AI agents and investors are also a key audience.
I hear the point you’re making. But I guess it’s not clear to me who the key audience of the model cards is? The red-teaming demonstrations in them (e.g., blackmailing) are exceptionally valuable for the safety community and as ways to engage the general public. It’s not obvious to me that these are directed at the government, investors or companies who want to switch to AI agents?
From the safety community it would be a very strange thing to say something like “Companies should only tell us about safety problems they are having when they can provide extraordinary proof that they have safety problems.”
That’s not how you handle companies dealing with product safety in any domain. You want companies to share information about safety issues even if the evidence for those safety issues is not overwhelming.