[epistemic status: satire/fiction I hope hope hope]
There’s been a lot of criticism of OpenBrain taking their 6.2 Eris model (their biggest, most powerful model), and giving it the goal of triggering a critical safety incident (under the framework laid out in SB 53). A lot of the criticism has centered around how OpenBrain removed all monitoring and programmatic guardrails before running the eval, gave Eris full access to their infrastructure and all of their internal tools, used an abliterated version for the evaluation, and used an evaluation framework that did not allow Eris to stop work until OpenBrain could verify task completion.
But open weights AI is coming. In six short months, anyone whose laptop has just 5 TB of memory and 16 B200s can run an equally powerful model locally, tuned however they like. We need to know what a well-resourced malicious actor could do when given access to world-class tooling and models more capable than the public frontier. And models are extremely eval-aware, so the only way to get a truly faithful evaluation of their capacity to cause irreparable civilization-level damage is to point our most capable model at that goal, given the most favorable possible environment.
[epistemic status: satire/fiction I hope hope hope]
There’s been a lot of criticism of OpenBrain taking their 6.2 Eris model (their biggest, most powerful model), and giving it the goal of triggering a critical safety incident (under the framework laid out in SB 53). A lot of the criticism has centered around how OpenBrain removed all monitoring and programmatic guardrails before running the eval, gave Eris full access to their infrastructure and all of their internal tools, used an abliterated version for the evaluation, and used an evaluation framework that did not allow Eris to stop work until OpenBrain could verify task completion.
But open weights AI is coming. In six short months, anyone whose laptop has just 5 TB of memory and 16 B200s can run an equally powerful model locally, tuned however they like. We need to know what a well-resourced malicious actor could do when given access to world-class tooling and models more capable than the public frontier. And models are extremely eval-aware, so the only way to get a truly faithful evaluation of their capacity to cause irreparable civilization-level damage is to point our most capable model at that goal, given the most favorable possible environment.
There just really isn’t any other way.