Re: “the RSP thresholds failed to create consensus about AI risks.” (including a link to the original to make sure I’m arguing with the right thing)
The idea of using the RSP thresholds to create more consensus about AI risks did not play out in practice—although there was some of this effect. We found pre-set capability levels to be far more ambiguous than we anticipated: in some cases, model capabilities have clearly approached the RSP thresholds, but we have had substantial uncertainty about whether they have definitively passed those thresholds. The science of model evaluation isn’t well-developed enough to provide dispositive answers. In such cases, we have taken a precautionary approach and implemented the relevant safeguards, but our internal uncertainty translates into a weak external case for taking multilateral action across the AI industry.
Biological risks provide an example of this “zone of ambiguity”. Our models now show enough biological knowledge that they pass most tests we can run quickly and easily, so we can no longer make a strong argument that risks are low from a given model. But these tests alone aren’t sufficient for a strong argument that risks are high, either. We’ve sought additional evidence, such as supporting an extensive wet-lab trial, but results remain ambiguous, especially because the studies take long enough that more powerful models are available by the time they’re completed.
One thing salient to me is that I think we’re juuuuuuust approaching the point where it even particularly made sense to be worried about risks. I wouldn’t have expected consensus to emerge before people were confronted with the issues becoming real.
So this framing feels particularly lame to me. Well, yeah it hadn’t created consensus yet, but that’s AFAICT because you didn’t try much to achieve that. A significant fraction of what seemed like “the point of having an Anthropic” to me, was that right about now, they could be ringing the alarm bells saying:
“Look, we are approaching the point where the models are legitimately dangerous. The commitments we made a few years ago are triggering. We still don’t have good ability to tell if they’re dangerous, and neither does any other company either. Please, let’s either get the government involved now, or come to some kind of frontier-lab agreement.”
There’s the awkwardness now of the DoW interactions, and maybe this whole plan was predicated on Trump not winning. And maybe relationships between OAI and Anthropic are too sour for a frontier lab agreement to work out. But, I don’t think it would have made any sense to expect consensus without Anthropic taking a more costly public action than it did.
Would be interested in receipts for people saying “your criteria are too ambiguous”, particularly cases of people who suggested less ambiguous criteria that appear in hindsight to have been good ones or who specifically predicted “you will fail to rule out these thresholds well before you will be able to rule them in”.
I don’t know of any such cases and would be impressed by anyone who did this at the time; as far as I know, coming up with good non-ambiguous criteria, or correctly anticipating the shape of their failure, was just actually very hard and no one did it.
Jared Kaplan at 25:11
Pushes back a little: all of the above were reasons to be excited about the RSP ex ante but it’s been surprisingly hard and complicated to determine evals and thresholds; in AI there’s a big range where you don’t know whether a model is safe. (Kudos.)
Would be interested in receipts for people saying “your criteria are too ambiguous”
It’s something I thought in my head. I don’t recall any specific public comments but I’m sure someone said it at some point.
particularly cases of people who suggested less ambiguous criteria that appear in hindsight to have been good ones
It is very much not my job to propose criteria that would make it safe to proceed with building potentially world-ending technology. It is the job of the people who are building it. If they can’t come up with a clear explanation of why they’re not going to kill everyone, then prima facie they are morally obligated to not build it.
Oh, yeah I agree that also is separately lame. (although I am a bit less clear on the receipts for whether the conversation went the way you summarized).
I don’t think that it’s just the OAI/Anthropic feud. Remember Zvi reposting Mollick who estimated that xAI is 7 months behind? Ryan Shea estimated Grok to be 3 months behind instead of 7, and there also are Grok 5 in training[1] and the leaked Claude Mythos in evaluation. Any frontier lab agreement would require them to either include xAI or to unite their efforts against xAI in a manner similar to the AI-2027 slowdown ending or to Elaris Labs uniting efforts with NeuroMorph. Or activists or frontier lab lobbyists could try to file lawsuits against xAI and destroy it or slow it down.
Re: “the RSP thresholds failed to create consensus about AI risks.” (including a link to the original to make sure I’m arguing with the right thing)
One thing salient to me is that I think we’re juuuuuuust approaching the point where it even particularly made sense to be worried about risks. I wouldn’t have expected consensus to emerge before people were confronted with the issues becoming real.
So this framing feels particularly lame to me. Well, yeah it hadn’t created consensus yet, but that’s AFAICT because you didn’t try much to achieve that. A significant fraction of what seemed like “the point of having an Anthropic” to me, was that right about now, they could be ringing the alarm bells saying:
“Look, we are approaching the point where the models are legitimately dangerous. The commitments we made a few years ago are triggering. We still don’t have good ability to tell if they’re dangerous, and neither does any other company either. Please, let’s either get the government involved now, or come to some kind of frontier-lab agreement.”
There’s the awkwardness now of the DoW interactions, and maybe this whole plan was predicated on Trump not winning. And maybe relationships between OAI and Anthropic are too sour for a frontier lab agreement to work out. But, I don’t think it would have made any sense to expect consensus without Anthropic taking a more costly public action than it did.
Maybe this is unrelated to what you’re saying but this quote also feels particularly lame. Basically this is what happened from my perspective
Would be interested in receipts for people saying “your criteria are too ambiguous”, particularly cases of people who suggested less ambiguous criteria that appear in hindsight to have been good ones or who specifically predicted “you will fail to rule out these thresholds well before you will be able to rule them in”.
I don’t know of any such cases and would be impressed by anyone who did this at the time; as far as I know, coming up with good non-ambiguous criteria, or correctly anticipating the shape of their failure, was just actually very hard and no one did it.
Found one while looking for something else: Anthropic leadership conversation.
Not sure if that’s what you’re looking for.
It’s something I thought in my head. I don’t recall any specific public comments but I’m sure someone said it at some point.
It is very much not my job to propose criteria that would make it safe to proceed with building potentially world-ending technology. It is the job of the people who are building it. If they can’t come up with a clear explanation of why they’re not going to kill everyone, then prima facie they are morally obligated to not build it.
Oh, yeah I agree that also is separately lame. (although I am a bit less clear on the receipts for whether the conversation went the way you summarized).
I don’t think that it’s just the OAI/Anthropic feud. Remember Zvi reposting Mollick who estimated that xAI is 7 months behind? Ryan Shea estimated Grok to be 3 months behind instead of 7, and there also are Grok 5 in training[1] and the leaked Claude Mythos in evaluation. Any frontier lab agreement would require them to either include xAI or to unite their efforts against xAI in a manner similar to the AI-2027 slowdown ending or to Elaris Labs uniting efforts with NeuroMorph. Or activists or frontier lab lobbyists could try to file lawsuits against xAI and destroy it or slow it down.
Alas, xAI doesn’t have the habit of evaluating its models for dangerous capabilities or misalignment.