As someone currently at an AI lab (though certainly disproportionately LW-leaning from within that cluster), my stance respectively would be
“AI development should stop entirely” oh man depends exactly how you operationalize it. I’d likely push a button that magically stopped it for 10 years, maybe for 100, probably not for all time though I don’t think the latter would be totally crazy. None of said buttons would be my ideal policy proposal. In all cases the decisionmaking is motivated by downstream effects on the long-run quality of the future, not on mundane benefits or company revenue or whatever.
“risks are so severe that no level of benefit justifies them” nah, I like my VNM continuity axiom thank you very much, no ontologically incommensurate outcomes for me. I do think they’re severe enough that benefits on the order of “guaranteed worldwide paradise for a million years for every living human” don’t justify increasing them by 10% though!
“the people currently working on AI are not the right people to be making these decisions” absolutely. Many specific alternative decisionmakers would be worse but I don’t think the current setup is anything like optimal.
“traditional political processes might be better equipped to govern AI development than the informal governance of the research community” Since ‘might’ is a very weak word I obviously agree with this. Do I think it’s more likely than not, idk, it’ll depend on your operationalization but probably. I do think there are non-consequentialist (and second-order consequentialist) reasons to default in favor of existing legitimate forms of government for this kind of decisionmaking, so it isn’t just a question of who is better equipped in a magic hypothetical where you perfectly transfer the power.
I don’t think my opinions on any of these topics are particularly rare among my coworkers either, and indeed you can see some opinions of this shape expressed in public by Anthropic very recently! Quoting from the constitution or the adolescence of technology I think there’s quite a lot in the theme of the third and fourth supposedly-unspeakable thoughts from the essay:
Claude should generally try to preserve functioning societal structures, democratic institutions, and human oversight mechanisms
We also want to be clear that we think a wiser and more coordinated civilization would likely be approaching the development of advanced AI quite differently—with more caution, less commercial pressure, and more careful attention to the moral status of AI systems. [...] we are not creating Claude the way an idealized actor would in an idealized world
Claude should refuse to assist with actions that would help concentrate power in illegitimate ways. This is true even if the request comes from Anthropic itself. [...] we want Claude to be cognizant of the risks this kind of power concentration implies, to view contributing to it as a serious harm that requires a very high bar of justification, and to attend closely to the legitimacy of the process and of the actors so empowered.
It is somewhat awkward to say this as the CEO of an AI company, but I think the next tier of risk [for seizing power] is actually AI companies themselves. [...] The main thing they lack is the legitimacy and infrastructure of a state [...] I think the governance of AI companies deserves a lot of scrutiny.
“risks are so severe that no level of benefit justifies them” nah, I like my VNM continuity axiom thank you very much, no ontologically incommensurate outcomes for me. I do think they’re severe enough that benefits on the order of “guaranteed worldwide paradise for a million years for every living human” don’t justify increasing them by 10% though!
What about… a hundred million years? What does your risk/benefit mapping actually look like?
As someone currently at an AI lab (though certainly disproportionately LW-leaning from within that cluster), my stance respectively would be
“AI development should stop entirely” oh man depends exactly how you operationalize it. I’d likely push a button that magically stopped it for 10 years, maybe for 100, probably not for all time though I don’t think the latter would be totally crazy. None of said buttons would be my ideal policy proposal. In all cases the decisionmaking is motivated by downstream effects on the long-run quality of the future, not on mundane benefits or company revenue or whatever.
“risks are so severe that no level of benefit justifies them” nah, I like my VNM continuity axiom thank you very much, no ontologically incommensurate outcomes for me. I do think they’re severe enough that benefits on the order of “guaranteed worldwide paradise for a million years for every living human” don’t justify increasing them by 10% though!
“the people currently working on AI are not the right people to be making these decisions” absolutely. Many specific alternative decisionmakers would be worse but I don’t think the current setup is anything like optimal.
“traditional political processes might be better equipped to govern AI development than the informal governance of the research community” Since ‘might’ is a very weak word I obviously agree with this. Do I think it’s more likely than not, idk, it’ll depend on your operationalization but probably. I do think there are non-consequentialist (and second-order consequentialist) reasons to default in favor of existing legitimate forms of government for this kind of decisionmaking, so it isn’t just a question of who is better equipped in a magic hypothetical where you perfectly transfer the power.
I don’t think my opinions on any of these topics are particularly rare among my coworkers either, and indeed you can see some opinions of this shape expressed in public by Anthropic very recently! Quoting from the constitution or the adolescence of technology I think there’s quite a lot in the theme of the third and fourth supposedly-unspeakable thoughts from the essay:
What about… a hundred million years? What does your risk/benefit mapping actually look like?