Thanks for the great comment. In my mind, the situation is something like Anthropic is pausing but also hoping to resume eventually, and continues its general approach of saying less rather than more about strategy and what’s going on internally. If that were the case, though, maybe they’d just not do further training runs internally without advertising this fact. One guess is preventing leaks is hard and safety-minded (quite senior) employees might be forcing action, though I admit the pause requires better motivation than the story provides.
Regarding communicable evidence: on the one hand, there are public statements about pausing going along with compelling shareable evidence, but on the other, I think historically it’s just been a pretty secretive company. Many of my conversations with Anthropic employees hit “I can’t talk about that.” It feels hard to see that changing.
Even in the story, it’s not clear the world is on track for meaningful global governance of the kind that would stop OAI/GDM though.
Completely. The point of the story was not to imply that the pausing results in everything working out great, merely that pausing would be a Highly Significant Event with Some Pretty Large Effects. Adequate legislation conditional on significant global legislation efforts feels like < 50% to me, at least.
Parts of the story that are now feeling implausible to me in the reaction are (a) the rapidity of reaction, especially the joint statement, which, if it did happen, would probably take much longer, (b) the conjunction of all those things happening – I think any of them individually or a subset is plausible.
My sense is that China mostly doesn’t believe in x-risk and mainly wants to be freed from US export controls so it can compete fairly,
That seems plausible. Mostly, it seems no one is trying to talk to China and take it for granted that they’ll want to compete and not take x-risk seriously enough to sign a treaty. I’m curious about your sources/evidence here.
Meanwhile, safety research at Anthropic basically collapses because they can no longer afford compute
I’m curious what the cost of ongoing safety research with existing models is if you’re just doing inference. My guess is not nearly as much as the big training runs? Other variables are how long do their products stay frontrunners, and are they profitable vs subsidized by investment still at this point. But that’d be like 6-12 months max I’m guessing. I don’t have the best models here.
Thanks for the great comment. In my mind, the situation is something like Anthropic is pausing but also hoping to resume eventually, and continues its general approach of saying less rather than more about strategy and what’s going on internally. If that were the case, though, maybe they’d just not do further training runs internally without advertising this fact. One guess is preventing leaks is hard and safety-minded (quite senior) employees might be forcing action, though I admit the pause requires better motivation than the story provides.
Regarding communicable evidence: on the one hand, there are public statements about pausing going along with compelling shareable evidence, but on the other, I think historically it’s just been a pretty secretive company. Many of my conversations with Anthropic employees hit “I can’t talk about that.” It feels hard to see that changing.
Completely. The point of the story was not to imply that the pausing results in everything working out great, merely that pausing would be a Highly Significant Event with Some Pretty Large Effects. Adequate legislation conditional on significant global legislation efforts feels like < 50% to me, at least.
Parts of the story that are now feeling implausible to me in the reaction are (a) the rapidity of reaction, especially the joint statement, which, if it did happen, would probably take much longer, (b) the conjunction of all those things happening – I think any of them individually or a subset is plausible.
That seems plausible. Mostly, it seems no one is trying to talk to China and take it for granted that they’ll want to compete and not take x-risk seriously enough to sign a treaty. I’m curious about your sources/evidence here.
I’m curious what the cost of ongoing safety research with existing models is if you’re just doing inference. My guess is not nearly as much as the big training runs? Other variables are how long do their products stay frontrunners, and are they profitable vs subsidized by investment still at this point. But that’d be like 6-12 months max I’m guessing. I don’t have the best models here.