Dario strongly implies that Anthropic “has this covered” and wouldn’t be imposing a massively unreasonable amount of risk if Anthropic proceeded as the leading AI company with a small buffer to spend on building powerful AI more carefully. I do not think Anthropic has this covered and in an (optimistic for Anthropic) world where Anthropic had a 3 month lead I think the chance of AI takeover would be high, perhaps around 20%.
I didn’t get this impression. (Or maybe I technically agree with your first sentence, if we remove the word “strongly”, but I think the focus on Anthropic being in the lead is weird and that there’s incorrect implicature from talking about total risk in the second sentence.)
As far as I can tell, the essay doesn’t talk much at all about the difference between Anthropic being 3 months ahead vs. 3 months behind.
“I believe the only solution is legislation” + “I am most worried about societal-level rules” and associated statements strongly imply that there’s significant total risk even if the leading company is responsible. (Or alternatively, that at some point, absent regulation, it will be impossible to be both in the lead and to take adequate precautions against risks.)
I do think the essay suggests that the main role of legislation is to (i) make the ‘least responsible players’ act roughly as responsibly as Anthropic, and (ii) to prevent the race & commercial pressures to heat up even further, which might make it “increasingly hard to focus on addressing autonomy risks” (thereby maybe forcing Anthropic to do less to reduce autonomy risks than they are now).
Which does suggest that, if Anthropic could keep spending their current amount of overhead on safety, then there wouldn’t be a huge amount of risks coming from Anthropic’s own models. And I would agree with you that this is very plausibly false, and that Anthropic will very plausibly be forced to either proceed in a way that creates a substantial risk of Claude taking over, or would have to massively increase their ratio of effort on safety vs. capabilities relative to where it is today. (In which case you’d want legislation to substantially reduce commercial pressures relative to where they are today, and not just make everyone invest about as much in safety as Anthropic is doing today.)
I didn’t get this impression. (Or maybe I technically agree with your first sentence, if we remove the word “strongly”, but I think the focus on Anthropic being in the lead is weird and that there’s incorrect implicature from talking about total risk in the second sentence.)
As far as I can tell, the essay doesn’t talk much at all about the difference between Anthropic being 3 months ahead vs. 3 months behind.
“I believe the only solution is legislation” + “I am most worried about societal-level rules” and associated statements strongly imply that there’s significant total risk even if the leading company is responsible. (Or alternatively, that at some point, absent regulation, it will be impossible to be both in the lead and to take adequate precautions against risks.)
I do think the essay suggests that the main role of legislation is to (i) make the ‘least responsible players’ act roughly as responsibly as Anthropic, and (ii) to prevent the race & commercial pressures to heat up even further, which might make it “increasingly hard to focus on addressing autonomy risks” (thereby maybe forcing Anthropic to do less to reduce autonomy risks than they are now).
Which does suggest that, if Anthropic could keep spending their current amount of overhead on safety, then there wouldn’t be a huge amount of risks coming from Anthropic’s own models. And I would agree with you that this is very plausibly false, and that Anthropic will very plausibly be forced to either proceed in a way that creates a substantial risk of Claude taking over, or would have to massively increase their ratio of effort on safety vs. capabilities relative to where it is today. (In which case you’d want legislation to substantially reduce commercial pressures relative to where they are today, and not just make everyone invest about as much in safety as Anthropic is doing today.)