I very much agree with you first point (see my comment here), I’m also somewhat sympathetic to your second point, but I’m a bit confused about the third.
Isn’t it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?
So is the only difference in Plan A that you lose locally-run open-weight models? I agree that’s a real loss, but my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those. Plus it seems pretty likely that competeitive open-weight development will stop anyway due to economic reasons and normal government regulations.
At least in Plan A there are some official efforts towards privacy-preserving technologies (though you are right that there is very little guarantee that nations don’t secretly pull out of the privacy-preserving rules). And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMDs. In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI. If that happens, you can also use the European AI if you care a lot about privacy.
Maybe I’m missing something, but Plan A doesn’t seem worse to me than the default trajectory from a privacy perspective.
my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those
First, one thing open weights do is not just let you run things locally as a consumer, but run weights as a smaller company to provide extra privacy. So open weights let a mid-size company run weights on their own servers, that they physically control, for instance. And they also let consumers choose to purchase tokens / inference from companies that try to maximize privacy—i.e., attempts like Tinfoil. (I think there are a few companies in this area, that’s just what my first search turned up.)
Second, even if a majority of users don’t actually use this, it still plausibly provides value for everyone because it’s a kind of herd immunity. Or at least, that’s the standard argument for why the parallel case of strong encryption is valuable, even though the vast majority of people don’t use it: if at least some reasonable % of people use strong encryption, then it means that (1) government cannot specifically target those who use it as a strong signal they are bad, and (2) you have the option of switching to it, if you find yourself being the kind of person who might be surveilled or targeted by the government. And some similar argument about the optionality provided by secure inference still seems go through.
And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMD.… In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI.
I gotta say that choosing between the regulators of the EU, China and the US to satisfy values specifically along the axis of privacy does not particularly fill me with joy :).
But seriously, I mean I don’t expect the kind of secure private inference Europe sets up for its governments to be available to foreign citizens, and I don’t think EU / China / US are the kind of national entities that sufficiently value privacy such that they’d compete to provide this value. Much much better situated for misc startups to try to provide it, I think.
Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)
I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.
But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.
This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”
Isn’t it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?
The US government can compel the disclosure of all recorded LLM conversations that it has a good reason to suspect contain evidence of wrongdoing, in accordance with longstanding norms about compelling companies to turn over evidence of wrongdoing. In theory, the US government can lie about its reasons for wanting access to any LLM conversation, but it is a substantial escalation that Plan A grants them hardware access to data which could previously only be obtained by lying to a third party.
I very much agree with you first point (see my comment here), I’m also somewhat sympathetic to your second point, but I’m a bit confused about the third.
Isn’t it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?
So is the only difference in Plan A that you lose locally-run open-weight models? I agree that’s a real loss, but my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those. Plus it seems pretty likely that competeitive open-weight development will stop anyway due to economic reasons and normal government regulations.
At least in Plan A there are some official efforts towards privacy-preserving technologies (though you are right that there is very little guarantee that nations don’t secretly pull out of the privacy-preserving rules). And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMDs. In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI. If that happens, you can also use the European AI if you care a lot about privacy.
Maybe I’m missing something, but Plan A doesn’t seem worse to me than the default trajectory from a privacy perspective.
First, one thing open weights do is not just let you run things locally as a consumer, but run weights as a smaller company to provide extra privacy. So open weights let a mid-size company run weights on their own servers, that they physically control, for instance. And they also let consumers choose to purchase tokens / inference from companies that try to maximize privacy—i.e., attempts like Tinfoil. (I think there are a few companies in this area, that’s just what my first search turned up.)
Second, even if a majority of users don’t actually use this, it still plausibly provides value for everyone because it’s a kind of herd immunity. Or at least, that’s the standard argument for why the parallel case of strong encryption is valuable, even though the vast majority of people don’t use it: if at least some reasonable % of people use strong encryption, then it means that (1) government cannot specifically target those who use it as a strong signal they are bad, and (2) you have the option of switching to it, if you find yourself being the kind of person who might be surveilled or targeted by the government. And some similar argument about the optionality provided by secure inference still seems go through.
I gotta say that choosing between the regulators of the EU, China and the US to satisfy values specifically along the axis of privacy does not particularly fill me with joy :).
But seriously, I mean I don’t expect the kind of secure private inference Europe sets up for its governments to be available to foreign citizens, and I don’t think EU / China / US are the kind of national entities that sufficiently value privacy such that they’d compete to provide this value. Much much better situated for misc startups to try to provide it, I think.
Yeah, I agree that from a privacy perspective, it is a real loss to lose the open-weigh models. (Though I’m unsure how much I buy the argument that privacy-preserving options are super important even if the vast majority don’t use it. I haven’t thought about this too much.)
I didn’t mean it though that European governments will be the ones setting up the AIs. My understanding is that Plan A envisions AI still being provided by companies, and the number of near-frontier model providers being somewhat large in different jurisdictions, because full research transparency makes it relatively easy to catch up. And given that the companies can’t really compete on the capabilities of their models, they will need a different moat, and I think ensuring privacy is one of the most natural selling points they can offer. Jurisdictions will also be interested in their AI companies being successful, and they might also be interested in privacy for their own reasons (e.g. the example I gave of different European governments wanting to rely on a company’s services without any of the other governments spying on the company). I wouldn’t be surprised if we had companies in the Plan A world that are approximately as trustworthy on privacy as Tinfoil is now.
But most importantly, how realistic do you think the alternative is? This is not a rhetorical question, I’m actually curious how big chance you give that we will go through the intelligence explosion with locally hosted open-weight models being available throughout. It seems very unlikely to me, because I expect that governments will want to clamp down first on misuse then on autonomous AI criminals.
Like pretty low, tbh, I think we ban weights far before a hypothetical point at which they would best be banned, in a way that’s very negative for AI safety, CoP, etc.
But like, I don’t see why I should fold my hands and be like “Yes, I give up on this issue.” Like what the hell; should other parts of AI safety be like “Yeah well of course the NSA is going to want a super powerful AI of its own, ah well. Guess that’s what we gotta let them have it.” Why is this the kind of issue where it’s correct to fold instead of fight?
No, I don’t think you need to fold. I just felt confused that your framing implied as if Plan A was uniquely bad for privacy, and it was not clear from your writing that it’s primarily about banning open-weight models. I wouldn’t have objected if you phrased things like “Plan A supports banning open-weight models. This is something that’s likely to happen on the default trajectory too, but I think this is very bad for privacy and for many other reasons, and I’m unhappy that Plan A supports this too.”
The US government can compel the disclosure of all recorded LLM conversations that it has a good reason to suspect contain evidence of wrongdoing, in accordance with longstanding norms about compelling companies to turn over evidence of wrongdoing. In theory, the US government can lie about its reasons for wanting access to any LLM conversation, but it is a substantial escalation that Plan A grants them hardware access to data which could previously only be obtained by lying to a third party.