I have plenty of complaints about this piece and wish Dario’s worldview/his-publicly-presented-stances were different.
But, holding those constant, overall I’m glad he wrote this. I’m glad autonomy risks are listed early on. One of my main hopes this year was for Dario and Demis to do more public advocacy in the sort of directions this points.
I also just… find myself liking some of the poetry of the section names. (I found the “Black Seas of Infinity” reference particularly satisfying)
Vaguely attempting to separate out “reasonable differences in worldview” from “feels kinda skeezy”:
Skeezy
The way this conflates religious/totalizing-orientation-to-AI pessismism with “it just seems pretty likely for AI to be massively harmful”. (I do think it’s fair to critique a kind of apocalyptic vibe that some folk have, although I think there’s also kind of similarly badly totalizing views of “AI will be our salvation/next-phase of evolution”, and if you’re going to bother critiquing that you should be addressing both)
That feels pretty obviously like a political move to try to position Anthropic as “a reasonable middle ground.” (I don’t strongly object to them pulling that move. But, I think there are better ways to pull it)
Disagreement
Misuse/Bad-Actors. I have some genuine uncertainty whether it makes sense to be as worried about misuse as Dario is. Most of my beliefs are of the form “misalignment is real bad and real difficult” so I’m not too worried about bad actors getting AI, but, it’s plausible that if we solved misalignment, bad actors would immediately become a problem and it’s right to be concerned about it.
Unclear about skeezy vs just disagreeing
His frame around regulation, and it not-being-possible-to-slow-down feels pretty self serving, and/or confusing.
I agree with his caution about regulating things we don’t understand yet. I might agree with the sentence “regulations should be as surgical as possible” (mostly because I think that’s usually true of regulations). But I don’t really see a regime where the regulations are not relatively extreme in some ways, and I think surgical implies something like “precise” and “minimal”.
I find it quite weird that he doesn’t explore at all the options for controlled takeoff. It sounds like he thinks… like, do export controls and a few simple trade-embargo things are the only way to slow down autocracies, and it’s important to beat autocracies, and therefore we can only potentially slow down a teeny amount.
The options to slowing down are all potentially somewhat crazy or intense (not like “Dyson Spheres” crazy, but, like, “go to war” level crazy), and I dunno if he’s just not saying them because he doesn’t want to say anything too intense sounding, or he honestly doesn’t think they’ll work.
He reads something like “negative-utilitarian for accidentally doing costly regulations.”
...
This document is clearly overall a kind of political document (trying to shape the zeitgeist) and I don’t have that strong a take about what sort of political documents are good to write. But, in a world where political discourse was overall better, I’d have liked if he included notions of what would change his mind about the general vibe of “the way out of this situation is through it, rather than via slowdown/stopping.” If you’re going to be one of the billion dollar companies hurtling us towards unprecedented challenges, with some reasons for thinking that’s correct, I think you should at least spell out the circumstances where you’d change your mind or stop or naturally pivot your strategy.
I have plenty of complaints about this piece and wish Dario’s worldview/his-publicly-presented-stances were different.
But, holding those constant, overall I’m glad he wrote this. I’m glad autonomy risks are listed early on. One of my main hopes this year was for Dario and Demis to do more public advocacy in the sort of directions this points.
I also just… find myself liking some of the poetry of the section names. (I found the “Black Seas of Infinity” reference particularly satisfying)
Vaguely attempting to separate out “reasonable differences in worldview” from “feels kinda skeezy”:
Skeezy
The way this conflates religious/totalizing-orientation-to-AI pessismism with “it just seems pretty likely for AI to be massively harmful”. (I do think it’s fair to critique a kind of apocalyptic vibe that some folk have, although I think there’s also kind of similarly badly totalizing views of “AI will be our salvation/next-phase of evolution”, and if you’re going to bother critiquing that you should be addressing both)
That feels pretty obviously like a political move to try to position Anthropic as “a reasonable middle ground.” (I don’t strongly object to them pulling that move. But, I think there are better ways to pull it)
Disagreement
Misuse/Bad-Actors. I have some genuine uncertainty whether it makes sense to be as worried about misuse as Dario is. Most of my beliefs are of the form “misalignment is real bad and real difficult” so I’m not too worried about bad actors getting AI, but, it’s plausible that if we solved misalignment, bad actors would immediately become a problem and it’s right to be concerned about it.
Unclear about skeezy vs just disagreeing
His frame around regulation, and it not-being-possible-to-slow-down feels pretty self serving, and/or confusing.
I agree with his caution about regulating things we don’t understand yet. I might agree with the sentence “regulations should be as surgical as possible” (mostly because I think that’s usually true of regulations). But I don’t really see a regime where the regulations are not relatively extreme in some ways, and I think surgical implies something like “precise” and “minimal”.
I find it quite weird that he doesn’t explore at all the options for controlled takeoff. It sounds like he thinks… like, do export controls and a few simple trade-embargo things are the only way to slow down autocracies, and it’s important to beat autocracies, and therefore we can only potentially slow down a teeny amount.
The options to slowing down are all potentially somewhat crazy or intense (not like “Dyson Spheres” crazy, but, like, “go to war” level crazy), and I dunno if he’s just not saying them because he doesn’t want to say anything too intense sounding, or he honestly doesn’t think they’ll work.
He reads something like “negative-utilitarian for accidentally doing costly regulations.”
...
This document is clearly overall a kind of political document (trying to shape the zeitgeist) and I don’t have that strong a take about what sort of political documents are good to write. But, in a world where political discourse was overall better, I’d have liked if he included notions of what would change his mind about the general vibe of “the way out of this situation is through it, rather than via slowdown/stopping.” If you’re going to be one of the billion dollar companies hurtling us towards unprecedented challenges, with some reasons for thinking that’s correct, I think you should at least spell out the circumstances where you’d change your mind or stop or naturally pivot your strategy.