It seems to me that under ideal circumstances, once we think we’ve invented FAI, before we turn it on, we share the design with a lot of trustworthy people we think might be able to identify problems. I think it’s good to have the design be as secret as possible at that point, because that allows the trustworthy people to scrutinize it at their leisure. I do think the people involved in the design are liable to attract attention—keeping this “FAI review project” secret will be harder than keeping the design itself secret. (It’s easier to keep the design for the bomb secret than hide the fact that top physicists keep mysteriously disappearing.) And any purported FAI will likely come after a series of lesser systems with lucrative commercial applications used to fund the project, and those lucrative commercial applications are also liable to attract attention. So I think it’s strategically valuable to have the distance between published material and a possible FAI design be as large as possible. To me, the story of nuclear weapons is a story of how this is actually pretty hard even when well-resourced state actors try to do it.
Of course, that has to be weighed against the benefit of openness. How is openness helpful? Openness lets other researchers tell you if they think you’re pursuing a dangerous research direction, or if there are serious issues with the direction you’re pursuing which you are neglecting. Openness helps attract collaborators. Openness helps gain prestige. (I would argue that prestige is actually harmful because it’s better to keep a low profile, but I guess prestige is useful for obtaining required funding.) How else is openness helpful?
My suspicion is that those papers on Arxiv with 5 citations are mostly getting cited by people who already know the author, and the Arxiv publication isn’t actually doing much to attract collaboration. It feels to me like if our goal is to help researchers get feedback on their research direction or find collaborators, there are better ways to do this than encouraging them to publish their work. So if we could put mechanisms in place to achieve those goals, that could remove much of the motivation for openness, which would be a good thing in my view.
It seems to me that under ideal circumstances, once we think we’ve invented FAI, before we turn it on, we share the design with a lot of trustworthy people we think might be able to identify problems. I think it’s good to have the design be as secret as possible at that point, because that allows the trustworthy people to scrutinize it at their leisure. I do think the people involved in the design are liable to attract attention—keeping this “FAI review project” secret will be harder than keeping the design itself secret. (It’s easier to keep the design for the bomb secret than hide the fact that top physicists keep mysteriously disappearing.) And any purported FAI will likely come after a series of lesser systems with lucrative commercial applications used to fund the project, and those lucrative commercial applications are also liable to attract attention. So I think it’s strategically valuable to have the distance between published material and a possible FAI design be as large as possible. To me, the story of nuclear weapons is a story of how this is actually pretty hard even when well-resourced state actors try to do it.
Of course, that has to be weighed against the benefit of openness. How is openness helpful? Openness lets other researchers tell you if they think you’re pursuing a dangerous research direction, or if there are serious issues with the direction you’re pursuing which you are neglecting. Openness helps attract collaborators. Openness helps gain prestige. (I would argue that prestige is actually harmful because it’s better to keep a low profile, but I guess prestige is useful for obtaining required funding.) How else is openness helpful?
My suspicion is that those papers on Arxiv with 5 citations are mostly getting cited by people who already know the author, and the Arxiv publication isn’t actually doing much to attract collaboration. It feels to me like if our goal is to help researchers get feedback on their research direction or find collaborators, there are better ways to do this than encouraging them to publish their work. So if we could put mechanisms in place to achieve those goals, that could remove much of the motivation for openness, which would be a good thing in my view.