Nice. I’ve only read the quoted paragraphs from the main body:
Despite prohibitions on large-scale training, AIs could continue getting more capable via improvements to the algorithms and software used to train them. Therefore, the coalition adopts appropriate
restrictions on research that contributes to frontier AI development or research that could endanger the verification methods in the agreement. These restrictions would cover certain machine
learning-related research and may eventually expand to include other AI paradigms if those paradigms
seem likely to lead to ASI. Coalition members draw on their experience restricting research in other
dangerous fields such as nuclear and chemical weapons. These research restrictions aim to be as
narrow as possible, preventing the creation of more capable general AI models while still allowing
safe and beneficial narrow AI models to be created. The coalition encourages and makes explicit
carve-outs for safe, application-specific AI activities, such as self-driving cars and other uses that
provide benefits to society.
Verification that members are adhering to research restrictions is aided by the fact that relatively
few people have the skills to contribute to this research—likely only thousands or tens of thousands,
see Appendix A, Article IX. Nations verify compliance by way of intelligence gathering, interviews
with the researchers, whistleblower programs, and more. This verification aims to use non-invasive
measures to ensure that researchers are not working on restricted topics. Additionally, inspectors
verify that the medium-scale training allowed by the agreement uses only approved methods (e.g.,
from a Whitelist) and does not make use of any novel AI methods or algorithms; these would be
evidence that restricted research had taken place.
My low-effort attempted very short summary of this aspect of the treaty is:
Part of the treaty sets up a committee that decides what counts as AGI research. That stuff is banned.
Is that basically right? (And there’s more discussion of how you’d enforce that, and a few examples of non-AGI research are given, and some comparisons with nuclear bombs and Asilomar are given.)
I don’t have a better idea, so this seems as promising as I can think of. I’m curious though for ideas about
Some sort of category that could actually work here.
You need a category that’s politically viable and also captures most or all of AGI research.
It’s ok but sad and also maybe politically more difficult to exclude a bunch of stuff that isn’t AGI research.
For example (some bad ideas to spur better ideas from others):
You could say “you have to declare a specific task you’re trying to solve, like protein interactions or self-driving cars; the task can’t be super broad / heterogeneous, like predicting text”.
You could say “no running searches over very algorithmically rich spaces, or strongly enriching or speeding up existing searches”.
You could say “your system has to demonstrate by lesion experiments that it requires domain-specific knowledge”.
You could say “if other researchers can copy-paste your system into another domain, and it works well, it’s banned”.
You could say “if our red team is able to use your system to do X and Y and Z, then it’s banned”.
You could say “you have to state reasons why your research is interesting/promising, and those reasons have to sound like the promisingness comes from something domain-specific”.
And/or, some way of having really good governance.
It seems like if the committee goes awry, it’s very easy to pretend you’ve drawn a good boundary, but you haven’t.
It seems likely for the committee to go awry, given how much pressure there could be for less restriction.
Curious for any thoughts on how to do that, e.g. examples of governance with significant responsibility to develop unclear policies that went well despite strong pressures.
Hey Tsvi, thanks for the ideas! The short answer is that we don’t have good answers about what the details of Article VIII should be. It’s reasonably likely that I will work on this as my next big project and that I’ll spend a couple of months on it. If so, I’ll keep these ideas in mind—they seem like a reasonable first pass.
Nice. I’ve only read the quoted paragraphs from the main body:
And then skimmed the appendix “ARTICLE VIII — Restricted Research: AI Algorithms and Hardware”. I’m very happy to see any kind of proposal that includes a serious attempt to ban AGI research. I want to also mention “The Problem with Defining an “AGI Ban” by Outcome (a lawyer’s take).”.
My low-effort attempted very short summary of this aspect of the treaty is:
Is that basically right? (And there’s more discussion of how you’d enforce that, and a few examples of non-AGI research are given, and some comparisons with nuclear bombs and Asilomar are given.)
I don’t have a better idea, so this seems as promising as I can think of. I’m curious though for ideas about
Some sort of category that could actually work here.
You need a category that’s politically viable and also captures most or all of AGI research.
It’s ok but sad and also maybe politically more difficult to exclude a bunch of stuff that isn’t AGI research.
For example (some bad ideas to spur better ideas from others):
You could say “you have to declare a specific task you’re trying to solve, like protein interactions or self-driving cars; the task can’t be super broad / heterogeneous, like predicting text”.
You could say “no running searches over very algorithmically rich spaces, or strongly enriching or speeding up existing searches”.
You could say “your system has to demonstrate by lesion experiments that it requires domain-specific knowledge”.
You could say “if other researchers can copy-paste your system into another domain, and it works well, it’s banned”.
You could say “if our red team is able to use your system to do X and Y and Z, then it’s banned”.
You could say “you have to state reasons why your research is interesting/promising, and those reasons have to sound like the promisingness comes from something domain-specific”.
It might be possible to targetedly make some boundaries more coordinatable-on. Cf. https://tsvibt.blogspot.com/2025/11/constructing-and-coordinating-around.html
And/or, some way of having really good governance.
It seems like if the committee goes awry, it’s very easy to pretend you’ve drawn a good boundary, but you haven’t.
It seems likely for the committee to go awry, given how much pressure there could be for less restriction.
Curious for any thoughts on how to do that, e.g. examples of governance with significant responsibility to develop unclear policies that went well despite strong pressures.
Hey Tsvi, thanks for the ideas! The short answer is that we don’t have good answers about what the details of Article VIII should be. It’s reasonably likely that I will work on this as my next big project and that I’ll spend a couple of months on it. If so, I’ll keep these ideas in mind—they seem like a reasonable first pass.