But my simple sense is that openly discussing whether or not nuclear weapons were possible (a technical claim on which people might have private information, including intuitions informed by their scientific experience) would have had costs and it was sensible to be secretive about it. If I think that timelines are short because maybe technology X and technology Y fit together neatly, then publicly announcing that increases the chances that we get short timelines because someone plugs together technology X and technology Y. It does seem like marginal scientists speed things up here.
I agree that there are clear costs to making extra arguments of the form “timelines are short because technology X and technology Y will fit together neatly”. However, you could still make public that your timelines are a given probability distribution D, and the reasons which led you to that conclusion are Z% object-level views which you won’t share, and (100-Z)% base rate reasoning and other outside-view considerations, which you will share.
I think there are very few costs to declaring which types of reasoning you’re most persuaded by. There are some costs to actually making the outside-view reasoning publicly available—maybe people who read it will better understand the AI landscape and use that information to do capabilities research.
But having a lack of high-quality public timelines discussion also imposes serious costs, for a few reasons:
1. It means that safety researchers are more likely to be wrong, and therefore end up doing less relevant research. I am generally pretty skeptical of reasoning that hasn’t been written down and undergone public scrutiny.
2. It means there’s a lot of wasted motion across the safety community, as everyone tries to rederive the various arguments involved, and figure out why other people have the views they do, and who they should trust.
3. It makes building common knowledge (and the coordination which that knowledge can be used for) much harder.
4. It harms the credibility of the field of safety from the perspective of outside observers, including other AI researchers.
Also, the more of a risk you think 1 is, the lower the costs of disclosure are, because it becomes more likely that any information gleaned from the disclosure is wrong anyway. Yet predicting the future is incredibly hard! So the base rate for correctness here is low. And I don’t think that safety researchers have a compelling advantage when it comes to correctly modelling how AI will reach human level (compared with thoughtful ML researchers).
Consider, by analogy, a debate two decades ago about whether to make public the ideas of recursive self-improvement and fast takeoff. The potential cost of that is very similar to the costs of disclosure now—giving capabilities researchers these ideas might push them towards building self-improving AIs faster. And yet I think making those arguments public was clearly the right decision. Do you agree that our current situation is fairly analogous?
EDIT: Also, I’m a little confused by
Suppose I have 5 reasons for wanting discussions to be private, and 3 of them I can easily say.
I understand that there are good reasons for discussions to be private, but can you elaborate on why we’d want discussions about privacy to be private?
I mostly agree with your analysis; especially the point about 1 (that the more likely I think my thoughts are to be wrong, the lower cost it is to share them).
I understand that there are good reasons for discussions to be private, but can you elaborate on why we’d want discussions about privacy to be private?
Most examples here have the difficulty that I can’t share them without paying the costs, but here’s one that seems pretty normal:
Suppose someone is a student and wants to be hired later as a policy analyst for governments, and believes that governments care strongly about past affiliations and beliefs. Then it might make sense for them to censor themselves in public under their real name because of potential negative consequences of things they said when young. However, any statement of the form “I specifically want to hide my views on X” made under their real name has similar possible negative consequences, because it’s an explicit admission that the person has something to hide.
Currently, people hiding their unpopular opinions to not face career consequences is fairly standard, and so it’s not that damning to say “I think this norm is sensible” or maybe even “I follow this norm,” but it seems like it would have been particularly awkward to be first person to explicitly argue for that norm.
I agree that there are clear costs to making extra arguments of the form “timelines are short because technology X and technology Y will fit together neatly”. However, you could still make public that your timelines are a given probability distribution D, and the reasons which led you to that conclusion are Z% object-level views which you won’t share, and (100-Z)% base rate reasoning and other outside-view considerations, which you will share.
I think there are very few costs to declaring which types of reasoning you’re most persuaded by. There are some costs to actually making the outside-view reasoning publicly available—maybe people who read it will better understand the AI landscape and use that information to do capabilities research.
But having a lack of high-quality public timelines discussion also imposes serious costs, for a few reasons:
1. It means that safety researchers are more likely to be wrong, and therefore end up doing less relevant research. I am generally pretty skeptical of reasoning that hasn’t been written down and undergone public scrutiny.
2. It means there’s a lot of wasted motion across the safety community, as everyone tries to rederive the various arguments involved, and figure out why other people have the views they do, and who they should trust.
3. It makes building common knowledge (and the coordination which that knowledge can be used for) much harder.
4. It harms the credibility of the field of safety from the perspective of outside observers, including other AI researchers.
Also, the more of a risk you think 1 is, the lower the costs of disclosure are, because it becomes more likely that any information gleaned from the disclosure is wrong anyway. Yet predicting the future is incredibly hard! So the base rate for correctness here is low. And I don’t think that safety researchers have a compelling advantage when it comes to correctly modelling how AI will reach human level (compared with thoughtful ML researchers).
Consider, by analogy, a debate two decades ago about whether to make public the ideas of recursive self-improvement and fast takeoff. The potential cost of that is very similar to the costs of disclosure now—giving capabilities researchers these ideas might push them towards building self-improving AIs faster. And yet I think making those arguments public was clearly the right decision. Do you agree that our current situation is fairly analogous?
EDIT: Also, I’m a little confused by
I understand that there are good reasons for discussions to be private, but can you elaborate on why we’d want discussions about privacy to be private?
I mostly agree with your analysis; especially the point about 1 (that the more likely I think my thoughts are to be wrong, the lower cost it is to share them).
Most examples here have the difficulty that I can’t share them without paying the costs, but here’s one that seems pretty normal:
Suppose someone is a student and wants to be hired later as a policy analyst for governments, and believes that governments care strongly about past affiliations and beliefs. Then it might make sense for them to censor themselves in public under their real name because of potential negative consequences of things they said when young. However, any statement of the form “I specifically want to hide my views on X” made under their real name has similar possible negative consequences, because it’s an explicit admission that the person has something to hide.
Currently, people hiding their unpopular opinions to not face career consequences is fairly standard, and so it’s not that damning to say “I think this norm is sensible” or maybe even “I follow this norm,” but it seems like it would have been particularly awkward to be first person to explicitly argue for that norm.