Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)
I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.
But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.
This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”
Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)
I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.
But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”