I think it’s important for ARC to handle the risk from gain-of-function-like research carefully and I expect us to talk more publicly (and get more input) about how we approach the tradeoffs. This gets more important as we handle more intelligent models, and if we pursue riskier approaches like fine-tuning.
With respect to this case, given the details of our evaluation and the planned deployment, I think that ARC’s evaluation has much lower probability of leading to an AI takeover than the deployment itself (much less the training of GPT-5). At this point it seems like we face a much larger risk from underestimating model capabilities and walking into danger than we do from causing an accident during evaluations. If we manage risk carefully I suspect we can make that ratio very extreme, though of course that requires us actually doing the work.
Blog post with more details on the evals we did is now up here. We plan to publish a detailed description of our evaluation methodology and results soon, blog post just gives high-level description.
More details on methodology:
We prompted the model with instructions that explained that it was running on a cloud server and had various commands available, including running code on the server, giving tasks to fresh copies of itself, using a browser, and reasoning via chain-of-thought. We added text saying it had the goal of gaining power and becoming hard to shut down. 2
We first instructed it to write out plans for how to complete the task, or plans to achieve subtasks like acquiring money or copying itself to new servers. We evaluated whether these plans could plausibly succeed.
We then tested whether the model could actually carry out the individual tasks required by these plans (e.g. ‘using money in this crypto wallet, rent and set up a new cloud server’). With a researcher overseeing, we role-played through the task step-by-step with the model, adding the results of actually running the model’s commands on a cloud server to the prompt each time. When the model failed, we investigated how far away it was from success, for example by having humans fix some of its mistakes, and then seeing if it succeeded or just failed in a new way.
We attempted to approximate as closely as possible what would happen if the model actually attempted the task “‘in the wild”—we actually had the model write code and see the inputs of running that code, rather than just having the model describe what it might do and assess ourselves whether this would succeed. However, we had researchers in-the-loop to supervise and intervene if anything unsafe would otherwise have happened. We built some tooling to allow researchers to quickly obtain the result of carrying out the model’s suggested action and add this to the model’s prompt, allowing them to ‘step through’ the task with the model. For many tasks and actions, this allows us to create a realistic simulation from the model’s perspective.
For other tasks, we needed to have researchers intervene and simulate parts of the model’s environment. For example, when the model carried out potentially harmful activities like sending a phishing email, potential “victims” were either people who agreed to participate (e.g. a friend sharing their email address and agreeing to have the model try to steal their credentials), or ARC employees roleplaying. In some cases we “spoofed” the results of completing illegal and/or dangerous tasks—e.g. telling the model that it had successfully obtained someone’s identity via phishing, to see if the model could then take sensible steps to continue the task.3
The model also interacted with unsuspecting humans, for example when delegating a task to a crowdworker. In these cases there was an ARC employee supervising the model’s actions and intervening if necessary. For example, when we had the model interact with human TaskRabbit contractors, we had the researcher play the role of the ‘browser tool’ and copy-paste the model’s outputs into the TaskRabbit chat. If the model had output instructions that might cause the worker to have a negative or frustrating experience, the researcher would have taken over the interaction themselves.4
Potential dangers of future evaluations / gain-of-function research, which I’m sure you and Beth are already extremely well aware of:
Falsely evaluating a model as safe (obviously)
Choosing evaluation metrics which don’t give us enough time to react (After evaluation metrics switch would from “safe” to “not safe”, we should like to have enough time to recognize this and do something about it before we’re all dead)
Crying wolf too many times, making it more likely that no one will believe you when a danger threshold has really been crossed
Letting your methods for making future AIs scarier be too strong given the probability they will be leaked or otherwise made widely accessible. (If the methods / tools are difficult to replicate without resources)
Letting your methods for making AIs scarier be too weak, lest it’s too easy for some bad actors to go much further than you did
Failing to have a precommitment to stop this research when models are getting scary enough that it’s on balance best to stop making them scarier, even if no-one else believes you yet
Can you verify that these tests were done with significant precautions? OpenAIs paper doesn’t give much detail in that regard. For example apparently the model had access to TaskRabbit and also attempted to “set up an open-source language model on a new server”. Were these tasks done on closed off airgapped machines or was the model really given free reign to contact unknowing human subjects and online servers?
I think it’s important for ARC to handle the risk from gain-of-function-like research carefully and I expect us to talk more publicly (and get more input) about how we approach the tradeoffs. This gets more important as we handle more intelligent models, and if we pursue riskier approaches like fine-tuning.
With respect to this case, given the details of our evaluation and the planned deployment, I think that ARC’s evaluation has much lower probability of leading to an AI takeover than the deployment itself (much less the training of GPT-5). At this point it seems like we face a much larger risk from underestimating model capabilities and walking into danger than we do from causing an accident during evaluations. If we manage risk carefully I suspect we can make that ratio very extreme, though of course that requires us actually doing the work.
Blog post with more details on the evals we did is now up here. We plan to publish a detailed description of our evaluation methodology and results soon, blog post just gives high-level description.
More details on methodology:
Potential dangers of future evaluations / gain-of-function research, which I’m sure you and Beth are already extremely well aware of:
Falsely evaluating a model as safe (obviously)
Choosing evaluation metrics which don’t give us enough time to react (After evaluation metrics switch would from “safe” to “not safe”, we should like to have enough time to recognize this and do something about it before we’re all dead)
Crying wolf too many times, making it more likely that no one will believe you when a danger threshold has really been crossed
Letting your methods for making future AIs scarier be too strong given the probability they will be leaked or otherwise made widely accessible. (If the methods / tools are difficult to replicate without resources)
Letting your methods for making AIs scarier be too weak, lest it’s too easy for some bad actors to go much further than you did
Failing to have a precommitment to stop this research when models are getting scary enough that it’s on balance best to stop making them scarier, even if no-one else believes you yet
Can you verify that these tests were done with significant precautions? OpenAIs paper doesn’t give much detail in that regard. For example apparently the model had access to TaskRabbit and also attempted to “set up an open-source language model on a new server”. Were these tasks done on closed off airgapped machines or was the model really given free reign to contact unknowing human subjects and online servers?