Good point. I think compute providers can steal model weights, as I said at the top. I think they currently have more incentive to steal architecture and training algorithms, since those are easier to use without getting caught, so I focused on “algorithmic secrets.”
Separately, are Amazon and Google incentivized to steal architecture and training algorithms? Meh. I think it’s very unlikely, since even if they’re perfectly ruthless their reputation is very important to them (plus they care about some legal risks). I think habryka thinks it’s more likely than I do. This is relevant to Anthropic’s security prioritization — security from compute providers might not be among the lowest-hanging fruit. And I think Fabien thinks it’s relevant to ASL-3 compliance, and I agree that ASL-3 probably wasn’t written with insider threat from compute providers in mind. But I’m not sure it’s relevant to ASL-3 compliance? The ASL-3 standard doesn’t say that actors are only in scope if they seem incentivized to steal stuff; the scope is based on actors’ capabilities.
Good point. I think compute providers can steal model weights, as I said at the top. I think they currently have more incentive to steal architecture and training algorithms, since those are easier to use without getting caught, so I focused on “algorithmic secrets.”
Separately, are Amazon and Google incentivized to steal architecture and training algorithms? Meh. I think it’s very unlikely, since even if they’re perfectly ruthless their reputation is very important to them (plus they care about some legal risks). I think habryka thinks it’s more likely than I do. This is relevant to Anthropic’s security prioritization — security from compute providers might not be among the lowest-hanging fruit. And I think Fabien thinks it’s relevant to ASL-3 compliance, and I agree that ASL-3 probably wasn’t written with insider threat from compute providers in mind. But I’m not sure it’s relevant to ASL-3 compliance? The ASL-3 standard doesn’t say that actors are only in scope if they seem incentivized to steal stuff; the scope is based on actors’ capabilities.