Are you barred from saying anything about your level of access?
Some things I can probably share but I don’t remember the exact policy.
If labs denied you from getting access you needed currently, would you be able to raise the issue publicly?
At some point yes. We usually do this in the evaluation report for the model. I would guess that if the model were not announced yet we would wait until release for a public announcement and meanwhile complain at the lab about any regulations they’re in violation of, and if the lack of access means we can’t rule out imminent x-risk, we’d also tell the government and anyone relevant. But I develop eval methodology, not do evals myself, so this is just my guess.
Thanks! Yeah that was my understanding of the case during evaluation of an unreleased model, I meant more in this specific case where it’s a question of access without AI R&D safeguards to a publicly released model.
Well, it would depend on our agreement with the lab. It would be unreasonable for labs to prevent METR from publishing its work without a good reason, and the report will always say what version of the model we tested. If it were important, METR could demand that we publish our methodology beforehand for preregistration or something, which would be far more info than just whether we have railfree access.
In general, METR is not very adversarial with labs on an everyday level. As long as we have editorial control it’s not currently the bottleneck, so there are many affordances we have but don’t use, or could have but don’t ask for.
Some things I can probably share but I don’t remember the exact policy.
At some point yes. We usually do this in the evaluation report for the model. I would guess that if the model were not announced yet we would wait until release for a public announcement and meanwhile complain at the lab about any regulations they’re in violation of, and if the lack of access means we can’t rule out imminent x-risk, we’d also tell the government and anyone relevant. But I develop eval methodology, not do evals myself, so this is just my guess.
Thanks! Yeah that was my understanding of the case during evaluation of an unreleased model, I meant more in this specific case where it’s a question of access without AI R&D safeguards to a publicly released model.
Well, it would depend on our agreement with the lab. It would be unreasonable for labs to prevent METR from publishing its work without a good reason, and the report will always say what version of the model we tested. If it were important, METR could demand that we publish our methodology beforehand for preregistration or something, which would be far more info than just whether we have railfree access.
In general, METR is not very adversarial with labs on an everyday level. As long as we have editorial control it’s not currently the bottleneck, so there are many affordances we have but don’t use, or could have but don’t ask for.