If you give substantially less restricted access to people who have shown themselves “trustworthy”, you have to look at how “trustworthiness” is going to be assessed. In any realistic case, the answer will be that the people considered “trustworthy” will be those associated with, or at the very least vouched for by, already powerful “Establishment(TM)” organizations. That’s exactly how Glasswing did it, and it’s pretty much the essence of every proposal I’ve heard from Anthropic or anybody else.
You go from a bad situation where “The Labs” control the frontier models, but at least grant a reasonable degree of access to more or less anybody, to a worse situation where access fundamentally depends on existing power, and of course makes it easier to get more power. Power increases for the favored, who are already powerful, and decreases for everybody else, even if some of those excluded are also somewhat powerful at the moment.
How could it not make concentration worse?
As for whether you get enough safety to justify the concentration of power, well...
Say you’re worried about bioweapons uplift. The people with the most motivation, and arguably the most demonstrated propensity[1], to use bioweapons are governments, yet they and their contractors will have no trouble passing the vetting, at least if they’re the “right” governments.
Same for computer security[2]. Both public and private entities are in the business of breaking security at scale. Many of both will also pass… whereas many of their targets will not, and therefore won’t get any help in defending themselves.
Not to mention that even though the criteria tend to get cast in terms of organizations, it’s still individuals who formulate the actual queries, and organizations often don’t have as much control over their affiliates as they think they do. So you actually have two layers of failure: the organizations you trust may betray you, or the individuals inside them may betray both them and you.
You can probably exclude a huge number of obvious kooks using the kinds of filter they’ve been deploying, or indeed many kinds of filters. But even with AI uplift, obvious kooks have a relatively low chance of executing anything successfully. Actors capable enough to make effective use of what you offer are going to be a lot harder to catch. I’m not saying you’ll catch none at all, but you’ll do it at the cost of deterring a lot of legitimate defensive work, and that gets worse the tighter you try to make the net.
The basic problem is that you have no really reliable signal about who the “good guys” are. The people with the best chance of gaming any system you do set up are (a) the powerful (so you get further concentration), and (b) competent and motivated attackers. Defense is actively disadvantaged. If you’re an attacker, attack is what you do; you specialize in exactly the things being restricted. You’re motivated to either legitimately pass the checks, or find a way around them. On the other hand, defenders aren’t necessarily specialized that way, and often have other things to think about. They’re less likely to jump through hoops to get approved, and less likely still to put in the work to get around restrictions.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets. Which you can of course do better by just witdrawing it entirely.
Footnote added on edit: I feel like I should mention that nobody, government or otherwise, has actually show that much propensity to use bioweapons. That’s probably because they’re crummy weapons that you can’t target or control. But at least if you’re a government, you can maybe hope the plague will stay in the other government’s territory, which is geographically separate from yours. If you’re a non-government, you’re more likely to be physically colocated with your enemies. Even if you have some kind of obsession about race or whatever, your targeting gets much harder. I don’t think anybody, including LLMs, actually knows how to target a given race with a bioweapon, and even if you did, you’d be one mutation away from off-target effects. So you basically have to be a total lunatic to even try. Which, again, means your chances of succeeding are going to be bad, because lunacy tends to have far-reaching effects on competence.
Quite bluntly, have you actually read the proposal and compared it against our current situation? Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by… looping in trusted partners to double-check their safety results?
I am suggesting a short period of early release access. We have already seen from the Mythos release that this is actually sufficient to patch critical software and ensure the release is less publicly disruptive. I have no clue how a few months of early access results in a serious “imbalance of control”—at worst, you can trivially fix this by rotating what groups get early access, neh?
Equally, if you’re worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than “the entire population of Earth”? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
We have somehow survived the problem of vetting who the “good guys” are despite the attackers incentives for centuries. It’s the basic concept of security clearances.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets.
That is part of what I am suggesting, yes. Conversely, “withdrawing it entirely” gets you absolutely zero new information, so I’m not sure how that’s possibly a fair comparison.
Perhaps I failed to communicate something clearly, but this all reads like a ridiculous strawman of my actual proposal, which is simply “we should have independent organizations involved in mundane auditing, so that more people are aware of capabilities before they become public, and we can involve the relevant security teams”
P.S. Cyber has been a common professional term for decades so maybe save the linguistic snobbery for your own personal blog? I don’t see what purpose is served by spelling out your personal distaste with one particular word I used, and I struggle to read it as anything other than an insult when you say it “reeks of clueless”. You could have been polite, or just cut the footnote entirely.
Edited to Add: Okay, I reviewed my post and comment and I don’t mention “bioterrorism” anyway so I’m now extremely convinced you’ve confused my post for someone else’s—why are you writing an entire footnote about something I never brought up?
Quite bluntly, have you actually read the proposal and compared it against our current situation?
Yes. But I didn’t read it carefully enough, and I let myself mentally import aspects of actual Glasswing that you didn’t actually mention. I don’t think it makes a critical difference, though.
The first thing I let get me was that you talk about it much more as a test of the model than I think actual Glasswing was. I believe that Glasswing was much more about selectively letting people use the known capabilities, with basically no intention to treat the early access people as evaluators or approvers of the model itself. You seem to see your proposal as some of both, though. As long as there’s any component at all where the early access people get to use the capabilities any of their own purposes, at least some significant power concentration concern remains.
Second, I mentally included followon stuff that wasn’t strictly part of Glasswing, but that’s strongly associated with it in my mind. In Glasswing, the early access group got Mythos. Later, everybody else got Fable, which intentionally nerfs many of the capabilities of Mythos, to the point of making it basically unusable for a lot of people and applications. That nerfing seems to be meant to be permanent. You didn’t mention that, although I think it’s pretty reasonable to expect something like it to be part of any real implementation of what you propose. If so, that amplifies the power concentration issue.
Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by… looping in trusted partners to double-check their safety results?
No, I’m afraid of power concentrating in the people they “loop in”.
… and honestly I don’t care if their intentions are nefarious or not, only about the results.
I have no clue how a few months of early access results in a serious “imbalance of control”
If it’s just early access, with no greater limitations on what the public eventually gets than on what the early users get, then that reduces the power concentration concern… somewhat. That’s a pretty big assumption, though. Would you mean to commit to never reducing the general-release capabilities based on what the early access found?
Even assuming pure early access, though, a few months is still a significant fraction of a “model cycle”. It’s not an advantage you can ignore.
at worst, you can trivially fix this by rotating what groups get early access, neh?
I assume you’re not suggesting rotating among just anybody who shows up and asks, only rotating among those who meet some criteria for being “trusted partners”. In practice, that’s going to be a small set of similar entities with similar goals and outlooks, probably with their own web of cooperative relationships. So, no, that doesn’t fix it.
You might not even be able to identify enough “trusted partners” to let you rotate at all.
We have somehow survived the problem of vetting who the “good guys” ared despite the attackers incentives for centuries. It’s the basic concept of security clearances.
There haven’t been security clearances for centuries. But honestly security clearances have many of the same problems. Much of the clearance process is a conformism check; that concentrates access among people who support the existing system… to the point of being willing to swear to keep its secrets without knowing what those secrets may be. No, they don’t (or at least didn’t) demand that you have any particular positions on (most) specific political questions, but they demand things that correlate with all kinds of stuff.
The clearance system obviously does manage to exclude some risky people, even some who aren’t obvious flakes. I don’t know that anybody actually knows how successful it is. It’s a self-perpetuating thing based on tradition and “common sense”, not something you can actually do statistics on. And it’s embedded in a larger classification system, in which compartmentation and other ways of reducing the number of people with access are actually way more important than clearance.
To whatever degree it does work, it works by vastly more detailed and invasive individualized investigation than I think you’re suggesting. You don’t get a security clearance based on your organizational affiliation; you’re granted the affiliation contingent on the clearance. Would you even want to import that kind of rigor?
And it’s aimed at actors with affiliated with mostly known adversaries, or people who look easy to coerce. There are more obvious flags for those than for freelance omnicidal mania, random criminal tendencies, etc.
Equally, if you’re worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than “the entire population of Earth”? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
Doesn’t that translate to the benefit being entirely from absolute reduction in access, and not from the phased part?
I don’t see what purpose is served by spelling out your personal distaste with one particular word I used,
It had nothing to do with you personally using “cyber”. I flagged it because at this point, it’s starting to confuse people when I don’t use, so I’m starting to feel like I have to explain. This probably means it’s getting close to the point where I have to just plain give up.
I’m sorry about the “clueless Beltway arrivisme”. I can see where you could take it as pointed at you, enough to gloss over the “Beltway arrivisme” part. It’s not aimed at you or really anybody using “cyber” now. It’s aimed at the people who originally introduced “cyber”. Those people were, in fact, clueless Beltway arrivistes. I forget that not everybody has that history nowadays.
Okay, I reviewed my post and comment and I don’t mention “bioterrorism” anyway so I’m now extremely convinced you’ve confused my post for someone else’s—why are you writing an entire footnote about something I never brought up?
You brought up “mundane harms”. I used canonical examples of “mundane harms”… the ones that apparently most motivated the original Glasswing. The footnote is to explain that I am not personally suggesting that anybody is very motivated to use bioweapons.
If you give substantially less restricted access to people who have shown themselves “trustworthy”, you have to look at how “trustworthiness” is going to be assessed. In any realistic case, the answer will be that the people considered “trustworthy” will be those associated with, or at the very least vouched for by, already powerful “Establishment(TM)” organizations. That’s exactly how Glasswing did it, and it’s pretty much the essence of every proposal I’ve heard from Anthropic or anybody else.
You go from a bad situation where “The Labs” control the frontier models, but at least grant a reasonable degree of access to more or less anybody, to a worse situation where access fundamentally depends on existing power, and of course makes it easier to get more power. Power increases for the favored, who are already powerful, and decreases for everybody else, even if some of those excluded are also somewhat powerful at the moment.
How could it not make concentration worse?
As for whether you get enough safety to justify the concentration of power, well...
Say you’re worried about bioweapons uplift. The people with the most motivation, and arguably the most demonstrated propensity [1] , to use bioweapons are governments, yet they and their contractors will have no trouble passing the vetting, at least if they’re the “right” governments.
Same for computer security [2] . Both public and private entities are in the business of breaking security at scale. Many of both will also pass… whereas many of their targets will not, and therefore won’t get any help in defending themselves.
Not to mention that even though the criteria tend to get cast in terms of organizations, it’s still individuals who formulate the actual queries, and organizations often don’t have as much control over their affiliates as they think they do. So you actually have two layers of failure: the organizations you trust may betray you, or the individuals inside them may betray both them and you.
You can probably exclude a huge number of obvious kooks using the kinds of filter they’ve been deploying, or indeed many kinds of filters. But even with AI uplift, obvious kooks have a relatively low chance of executing anything successfully. Actors capable enough to make effective use of what you offer are going to be a lot harder to catch. I’m not saying you’ll catch none at all, but you’ll do it at the cost of deterring a lot of legitimate defensive work, and that gets worse the tighter you try to make the net.
The basic problem is that you have no really reliable signal about who the “good guys” are. The people with the best chance of gaming any system you do set up are (a) the powerful (so you get further concentration), and (b) competent and motivated attackers. Defense is actively disadvantaged. If you’re an attacker, attack is what you do; you specialize in exactly the things being restricted. You’re motivated to either legitimately pass the checks, or find a way around them. On the other hand, defenders aren’t necessarily specialized that way, and often have other things to think about. They’re less likely to jump through hoops to get approved, and less likely still to put in the work to get around restrictions.
If you get a net safety benefit, I think most of it will actually just come from reducing the absolute amount of use your model gets. Which you can of course do better by just witdrawing it entirely.
Footnote added on edit: I feel like I should mention that nobody, government or otherwise, has actually show that much propensity to use bioweapons. That’s probably because they’re crummy weapons that you can’t target or control. But at least if you’re a government, you can maybe hope the plague will stay in the other government’s territory, which is geographically separate from yours. If you’re a non-government, you’re more likely to be physically colocated with your enemies. Even if you have some kind of obsession about race or whatever, your targeting gets much harder. I don’t think anybody, including LLMs, actually knows how to target a given race with a bioweapon, and even if you did, you’d be one mutation away from off-target effects. So you basically have to be a total lunatic to even try. Which, again, means your chances of succeeding are going to be bad, because lunacy tends to have far-reaching effects on competence.
Sorry, I’m old enough that “cyber” still reeks to me of clueless Beltway arrivisme, and I won’t use it.
Quite bluntly, have you actually read the proposal and compared it against our current situation? Are you seriously worried Anthropic is going to continue their nefarious consolidation of power by… looping in trusted partners to double-check their safety results?
I am suggesting a short period of early release access. We have already seen from the Mythos release that this is actually sufficient to patch critical software and ensure the release is less publicly disruptive. I have no clue how a few months of early access results in a serious “imbalance of control”—at worst, you can trivially fix this by rotating what groups get early access, neh?
Equally, if you’re worried about betrayal, surely that makes it better to have a trusted group of a few hundred people, rather than “the entire population of Earth”? It is quite obviously easier to monitor a few dozen organizations for malicious usage and bad actors, versus the entire public population.
We have somehow survived the problem of vetting who the “good guys” are despite the attackers incentives for centuries. It’s the basic concept of security clearances.
That is part of what I am suggesting, yes. Conversely, “withdrawing it entirely” gets you absolutely zero new information, so I’m not sure how that’s possibly a fair comparison.
Perhaps I failed to communicate something clearly, but this all reads like a ridiculous strawman of my actual proposal, which is simply “we should have independent organizations involved in mundane auditing, so that more people are aware of capabilities before they become public, and we can involve the relevant security teams”
P.S. Cyber has been a common professional term for decades so maybe save the linguistic snobbery for your own personal blog? I don’t see what purpose is served by spelling out your personal distaste with one particular word I used, and I struggle to read it as anything other than an insult when you say it “reeks of clueless”. You could have been polite, or just cut the footnote entirely.
Edited to Add: Okay, I reviewed my post and comment and I don’t mention “bioterrorism” anyway so I’m now extremely convinced you’ve confused my post for someone else’s—why are you writing an entire footnote about something I never brought up?
Yes. But I didn’t read it carefully enough, and I let myself mentally import aspects of actual Glasswing that you didn’t actually mention. I don’t think it makes a critical difference, though.
The first thing I let get me was that you talk about it much more as a test of the model than I think actual Glasswing was. I believe that Glasswing was much more about selectively letting people use the known capabilities, with basically no intention to treat the early access people as evaluators or approvers of the model itself. You seem to see your proposal as some of both, though. As long as there’s any component at all where the early access people get to use the capabilities any of their own purposes, at least some significant power concentration concern remains.
Second, I mentally included followon stuff that wasn’t strictly part of Glasswing, but that’s strongly associated with it in my mind. In Glasswing, the early access group got Mythos. Later, everybody else got Fable, which intentionally nerfs many of the capabilities of Mythos, to the point of making it basically unusable for a lot of people and applications. That nerfing seems to be meant to be permanent. You didn’t mention that, although I think it’s pretty reasonable to expect something like it to be part of any real implementation of what you propose. If so, that amplifies the power concentration issue.
No, I’m afraid of power concentrating in the people they “loop in”.
… and honestly I don’t care if their intentions are nefarious or not, only about the results.
If it’s just early access, with no greater limitations on what the public eventually gets than on what the early users get, then that reduces the power concentration concern… somewhat. That’s a pretty big assumption, though. Would you mean to commit to never reducing the general-release capabilities based on what the early access found?
Even assuming pure early access, though, a few months is still a significant fraction of a “model cycle”. It’s not an advantage you can ignore.
I assume you’re not suggesting rotating among just anybody who shows up and asks, only rotating among those who meet some criteria for being “trusted partners”. In practice, that’s going to be a small set of similar entities with similar goals and outlooks, probably with their own web of cooperative relationships. So, no, that doesn’t fix it.
You might not even be able to identify enough “trusted partners” to let you rotate at all.
There haven’t been security clearances for centuries. But honestly security clearances have many of the same problems. Much of the clearance process is a conformism check; that concentrates access among people who support the existing system… to the point of being willing to swear to keep its secrets without knowing what those secrets may be. No, they don’t (or at least didn’t) demand that you have any particular positions on (most) specific political questions, but they demand things that correlate with all kinds of stuff.
The clearance system obviously does manage to exclude some risky people, even some who aren’t obvious flakes. I don’t know that anybody actually knows how successful it is. It’s a self-perpetuating thing based on tradition and “common sense”, not something you can actually do statistics on. And it’s embedded in a larger classification system, in which compartmentation and other ways of reducing the number of people with access are actually way more important than clearance.
To whatever degree it does work, it works by vastly more detailed and invasive individualized investigation than I think you’re suggesting. You don’t get a security clearance based on your organizational affiliation; you’re granted the affiliation contingent on the clearance. Would you even want to import that kind of rigor?
And it’s aimed at actors with affiliated with mostly known adversaries, or people who look easy to coerce. There are more obvious flags for those than for freelance omnicidal mania, random criminal tendencies, etc.
Doesn’t that translate to the benefit being entirely from absolute reduction in access, and not from the phased part?
It had nothing to do with you personally using “cyber”. I flagged it because at this point, it’s starting to confuse people when I don’t use, so I’m starting to feel like I have to explain. This probably means it’s getting close to the point where I have to just plain give up.
I’m sorry about the “clueless Beltway arrivisme”. I can see where you could take it as pointed at you, enough to gloss over the “Beltway arrivisme” part. It’s not aimed at you or really anybody using “cyber” now. It’s aimed at the people who originally introduced “cyber”. Those people were, in fact, clueless Beltway arrivistes. I forget that not everybody has that history nowadays.
You brought up “mundane harms”. I used canonical examples of “mundane harms”… the ones that apparently most motivated the original Glasswing. The footnote is to explain that I am not personally suggesting that anybody is very motivated to use bioweapons.