It seems like we’re coming up on a point where, if the world is to survive under plan A (as opposed to plan S), we need to be able to say “this far and no further” to AI development. One of the key obstacles is that there isn’t a good way of specifying the stopping point/ceiling. So far, all of the attempts I’ve seen have talked about this in terms of amount of compute: maximum datacenter size in FLOPS, maximum training run size in FLOPS, etc.
I don’t think that approach is going to work as-is, first of all because algorithmic progress is going to quickly lower the amount of compute required to be dangerous, and second of all because in order for that plan to work at all, we need to somehow exploit the spikiness of AI capabilities to get AIs that are disproportionately bad at long-horizon planning.
Some alternative metrics, which I think could be usefully worked into legislation or international agreements:
Maximum compute per unique prompt per human-operator checkin. In a chatbot-style interface, this would be a maximum turn length (summed over subagents). In an API context, people setting up LLM-using software would be required to inspect the results of a given API usage and system prompt after a certain number of tokens used, and again at increasingly-spaced checkpoints.
Maximum compute per human-supervisor minute: In a chatbot-style interface, the software would measure how long you spent reading outputs and writing prompts, multiply it by some amount, and limit the runtime (summed over subagents) accordingly.
Minimum supervisory compute resource fraction: Some minimum percentage of total compute is devoted specifically to supervisory processes, meaning models/code that looks at inputs, outputs, or internals to decide whether it’s policy-compliant. To count as supervisory compute, it has to be able to terminate a run, and can’t help with tasks (other than by injecting a small number of policy notices).
All of these would require a significant amount of expanded detail and supporting software to implement, and there would have to be exemptions (with alternate requirements such as sandboxing) for training and eval environments where the compute-to-human-availability ratio is skewed.
I don’t think that stacking requirements like this is remotely close to sufficient to address existential risk from AI. But, if humanity is determined to try to thread the needle, they might widen the needle slightly, and if we are in fact going to follow AI 2040′s Plan A, anything we can do to widen that needle is valuable.
I don’t think “amount of time a human is looking at the reasoning logs” is the sort of thing that could be usefully worked into an international agreement. It’s very easily gameable, for one.
It seems like we’re coming up on a point where, if the world is to survive under plan A (as opposed to plan S), we need to be able to say “this far and no further” to AI development. One of the key obstacles is that there isn’t a good way of specifying the stopping point/ceiling. So far, all of the attempts I’ve seen have talked about this in terms of amount of compute: maximum datacenter size in FLOPS, maximum training run size in FLOPS, etc.
I don’t think that approach is going to work as-is, first of all because algorithmic progress is going to quickly lower the amount of compute required to be dangerous, and second of all because in order for that plan to work at all, we need to somehow exploit the spikiness of AI capabilities to get AIs that are disproportionately bad at long-horizon planning.
Some alternative metrics, which I think could be usefully worked into legislation or international agreements:
Maximum compute per unique prompt per human-operator checkin. In a chatbot-style interface, this would be a maximum turn length (summed over subagents). In an API context, people setting up LLM-using software would be required to inspect the results of a given API usage and system prompt after a certain number of tokens used, and again at increasingly-spaced checkpoints.
Maximum compute per human-supervisor minute: In a chatbot-style interface, the software would measure how long you spent reading outputs and writing prompts, multiply it by some amount, and limit the runtime (summed over subagents) accordingly.
Minimum supervisory compute resource fraction: Some minimum percentage of total compute is devoted specifically to supervisory processes, meaning models/code that looks at inputs, outputs, or internals to decide whether it’s policy-compliant. To count as supervisory compute, it has to be able to terminate a run, and can’t help with tasks (other than by injecting a small number of policy notices).
All of these would require a significant amount of expanded detail and supporting software to implement, and there would have to be exemptions (with alternate requirements such as sandboxing) for training and eval environments where the compute-to-human-availability ratio is skewed.
I don’t think that stacking requirements like this is remotely close to sufficient to address existential risk from AI. But, if humanity is determined to try to thread the needle, they might widen the needle slightly, and if we are in fact going to follow AI 2040′s Plan A, anything we can do to widen that needle is valuable.
(X crosspost)
I don’t think “amount of time a human is looking at the reasoning logs” is the sort of thing that could be usefully worked into an international agreement. It’s very easily gameable, for one.