I’m not sure but the wording in their footnote 1 seems unusually careful:
We think it’s valuable for AI developers to be able to share specific technical details with third parties without this information being shared further, and it’s very reasonable for AI developers to review 3rd-party eval reports to ensure no accidental sharing of sensitive IP.
We had an informal understanding with OpenAI that their review was checking for confidentiality / IP issues, rather than approving conclusions about safety or risk. We did not make changes to conclusions, takeaways or tone (or any other changes we considered problematic) based on their review. We are able to freely publish parts of the evaluation that depended only on information that is now public.
However, we expect some readers will want us to note that OpenAI would have had the legal right to block us from sharing conclusions about risk that depended on non-public information. Given that, this evaluation shouldn’t be interpreted as robust formal oversight or accountability that the public can be relying on METR to provide.
That being said, we think this evaluation is an excellent step forward and we are very supportive of prototyping the mechanics and content of third-party evaluation setups without the additional friction of a formalized oversight relationship.
The part “We are able to freely publish parts of the evaluation that depended only on information that is now public.” might suggest ”… and we were not allowed to publish the other parts, unlike previously”. I could be reading too much into it.
I’m not sure but the wording in their footnote 1 seems unusually careful:
The part “We are able to freely publish parts of the evaluation that depended only on information that is now public.” might suggest ”… and we were not allowed to publish the other parts, unlike previously”. I could be reading too much into it.