This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”