This post summarizes two papers that provide models of why scientific research tends to be so open, and then applies it to the development of powerful AI systems. The first models science as a series of discoveries, in which the first academic group to reach a discovery gets all the credit for it. It shows that for a few different models of info-sharing, info-sharing helps everyone reach the discovery sooner, but doesn’t change the probabilities for who makes the discovery first (called _race-clinching probabilities_): as a result, sharing all information is a better strategy than sharing none (and is easier to coordinate on than the possibly-better strategy of sharing just some information).
However, this theorem doesn’t apply when info sharing compresses the discovery probabilities _unequally_ across actors: in this case, the race-clinching probabilities _do_ change, and the group whose probability would go down is instead incentivized to keep information secret (which then causes everyone else to keep their information secret). This could be good news: it suggests that actors are incentivized to share safety research (which probably doesn’t affect race-clinching probabilities) while keeping capabilities research secret (thereby leading to longer timelines).
The second paper assumes that scientists are competing to complete a k-stage project, and whenever they publish, they get credit for all the stages they completed that were not yet published by anyone else. It also assumes that earlier stages have a higher credit-to-difficulty ratio (where difficulty can be different across scientists). It finds that under this setting scientists are incentivized to publish whenever possible. For AI development, this seems not to be too relevant: we should expect that with powerful AI systems, most of the “credit” (profit) comes from the last few stages, where it is possible to deploy the AI system to earn money.
Planned opinion:
I enjoyed this post a lot; the question of openness in AI research is an important one, that depends both on the scientific community and industry practice. The scientific community is extremely open, and the second paper especially seems to capture well the reason why. In contrast industry is often more secret (plausibly due to <@patents@>(@Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter@)). To the extent that we would like to change one community in the direction of the other, a good first step is to understand their incentives so that we can try to then change those incentives.
Planned summary for the Alignment Newsletter:
Planned opinion: