I don’t find these arguments particularly convincing for outweighing something like (for example) METR being able to say whether they have access to a model without AI R&D restrictions. I could see a stronger case maybe for Cyber access?
Even if this were the case, METR is exceptionally good at preemptively guarding against having to glomarize, so I’m somewhat surprised.
I think it is clearly very bad if in this specific case 3rd party evaluators have barred themselves from publicly raising this issue.
Could you talk publicly about special model access at Apollo? My impression was no.
I do think that “labs don’t want other orgs to start badgering them for special access” is actually a convincing reason for METR to not disclose any special model access. I want third-party model access to be as cheap and riskless as possible for the labs! At least, in the current regime where we are relying solely on their goodwill.
I am confused why we are making labs arguments for them. I am also extremely unconvinced by this reason. The idea that safety focused orgs should preemptively avoid raising issues of model access in this specific case of a deployed model with specifc AI R&D safeguards seems to be pretty severe pessimization.
It’s difficult to quanitfy, but I’m pretty opposed to preemptive pressure to avoid raising this issue (or avoid mentioning it) just because “labs might get some more asks” (this is extremely, extremely cheap for them to say no to or ignore). This is also already routine in other aspects like evaluations. I’d even be more symapthetic to domains like cyber or bio. In this case though I feel like we’re optimizing against ourselves for no gain.
No, Apollo does not currently have access to Fable without the strict AI R&D classifiers applied.
Seems plausible that you’d wanna glomarise whether you have special model access. If people know you have special access then:
It makes you a target for hackers/spies who want those sweet API keys.
It might impose costs on the labs, e.g. other orgs start badgering them for special access, saying “Well, you gave this org access! No fair!”
I don’t find these arguments particularly convincing for outweighing something like (for example) METR being able to say whether they have access to a model without AI R&D restrictions. I could see a stronger case maybe for Cyber access?
Even if this were the case, METR is exceptionally good at preemptively guarding against having to glomarize, so I’m somewhat surprised.
I think it is clearly very bad if in this specific case 3rd party evaluators have barred themselves from publicly raising this issue.
Could you talk publicly about special model access at Apollo? My impression was no.
I do think that “labs don’t want other orgs to start badgering them for special access” is actually a convincing reason for METR to not disclose any special model access. I want third-party model access to be as cheap and riskless as possible for the labs! At least, in the current regime where we are relying solely on their goodwill.
I am confused why we are making labs arguments for them. I am also extremely unconvinced by this reason. The idea that safety focused orgs should preemptively avoid raising issues of model access in this specific case of a deployed model with specifc AI R&D safeguards seems to be pretty severe pessimization.
It’s difficult to quanitfy, but I’m pretty opposed to preemptive pressure to avoid raising this issue (or avoid mentioning it) just because “labs might get some more asks” (this is extremely, extremely cheap for them to say no to or ignore). This is also already routine in other aspects like evaluations. I’d even be more symapthetic to domains like cyber or bio. In this case though I feel like we’re optimizing against ourselves for no gain.
No, Apollo does not currently have access to Fable without the strict AI R&D classifiers applied.