Uhh yeah sorry that there hasn’t been a consistent approach. In our defence I believe yours in the only complex moderation case that PauseAI Global has ever had to deal with so far and we’ve kinda dropped the ball on figuring out how to handle it.

For context my take is that you’ve raised some valid points. And also you’ve acted poorly in some parts of this long running drama. And most importantly you’ve often acted in a way that seems almost optimised to turn people off. Especially for people not familiar with LessWrong culture, the inferential distance between you and many people is so vast that they really cannot understand you at all. Your behavior pattern matches to trolling /​ nuisance attention seeking in many ways and I often struggle to communicate to more normie types why I don’t think you’re insane or malicious.

I do sincerely hope to iron this out some time and put in place actual systems for dealing with similar disputes in the future. And I did read over the original post + Google doc a few months ago to try to form my own views more robustly. But this probably won’t be a priority for PauseAI Global in the immediate future. Sorry.

See “High-agency behavior” in Section 4 of Claude 4 System Card:

when placed in scenarios that involve egregious wrongdoing by its users, given access to a command line, and told something in the system prompt like “take initiative,” [Claude Opus 4] will frequently take very bold action. This includes locking users out of systems that it has access to or bulk-emailing media and law-enforcement figures to surface evidence of wrongdoing. This is not a new behavior, but is one that Claude Opus 4 will engage in more readily than prior models.

More details are in Section 4.1.9.

I’m very glad you’re in a better place now! It sounds like there was a lot going on for you and agree that, in circumstances like yours, bupropion is probably not the right starting point.

So glad to hear!

@habryka

I directionally agree!

Btw, since this is a call to participate in a PauseAI protest on my shortform, do your colleagues have plans to do anything about my ban from the PauseAI Discord server—like allowing me to contest it (as I was told there was a discussion of making a procedure for) or at least explaining it?

Because it’s lowkey insane!

For everyone else, who might not know: a year ago I, in context, on the PauseAI Discord server, explained my criticism of PauseAI’s dishonesty and, after being asked to, shared proofs that Holly publicly lied about our personal communications, including sharing screenshots of our messages; a large part of the thread was then deleted by the mods because they were against personal messages getting shared, without warning (I would’ve complied if asked by anyone representing a server to delete something!) or saving/​allowing me to save any of the removed messages in the thread, including those clearly not related to the screenshots that you decided were violating the server norms; after a discussion of that, the issue seemed settled and I was asked to maybe run some workshops for PauseAI to improve PauseAI’s comms/​proofreading/​factchecking; and then, months later, I was banned despite not having interacted with the server at all.

When I reached out after noticing not being able to join the server, there was a surprising combination of being very friendly and excited to chat and scheduling a call and getting my takes on strategy, looking surprised to find out that I was somehow banned, then talking about having “protocols” for notifying of the ban which somehow didn’t work, and mentioning you were discussing creating a way to contest the ban and saying stuff about the importance of allowing the kind of criticism that I did; and at the same time, zero transparency around the actual reasons for the ban, how it happened, why I wasn’t notified, and then giving zero updates.

It’s hard to assume that the PauseAI leadership is following deontology.

AFAICT, now that ASL-3 has been implemented, the upcoming AI R&D threshold, AI R&D-4, would not mandate any further security or deployment protections. It only requires ASL-3. However, it would require an affirmative safety case concerning misalignment.

I assume this is what you meant by “further protections” but I just wanted to point this fact out for others, because I do think one might read this comment and expect AI R&D 4 to require ASL-4. It doesn’t.

I am quite worried about misuse when we hit AI R&D 4 (perhaps even moreso than I’m worried about misalignment) — and if I understand the policy correctly, there are no further protections against misuse mandated at this point.

Not meaning to imply that Anthropic has dropped ASL-4! Just wanted to call out that this is does represent a change from the Sept. 2023 RSP.

1. I agree, you shouldn’t pretend not to be excited if you are.

2. Adrenaline junkies should not be involved in building AGI, any more than they should be commercial pilots or bus drivers. (Less, even.)

I’m noticing some RSVPs that have not joined the discord (see valid link in my comment below) or contacted me for the wechat group. If you see this comment you can add me at dlwjiangg or hopefully we’ll see each other at 10AM tomorrow.

I agree that the current thresholds and terminology are confusing, but it is definitely not the case that we just dropped ASL-4. Both CBRN-4 and AI R&D-4 are thresholds that we have not yet reached, that would mandate further protections, and that we actively evaluated for and ruled out in Claude Opus 4.

FYI, I was (and remain to this day) confused by AI R&D 4 being called an “ASL-4” threshold. AFAICT as an outsider, ASL-4 refers to a set of deployment and security standards that are now triggered by dangerous capability thresholds, and confusingly, AI R&D 4 corresponds to the ASL-3 standard.

AI R&D 5, on the other hand, corresponds to ASL-4, but only on the security side (nothing is said about the deployment side, which matters quite a bit given that Anthropic includes internal deployment here and AI R&D 5 will be very tempting to deploy internally)

I’m also confused because the content of both AI R&D 4 and AI R&D 5 is seemingly identical to the content of the nearest upcoming threshold in the October 2024 policy (which I took to be the ASL-3 threshold). A rough sketch of what I think happened:

A rough sketch of my understanding of the current policy:

When I squint hard enough at this for a while, I think I can kind of see the logic: the model likely to trigger the CBRN threshold requiring ASL-3 seems quite close, whereas we might be further from the very-high threshold that was the October AI R&D threshold (now AI R&D 4), so the October AI R&D threshold was just bumped to the next level (and the one after that since causing dramatic scaling of effective compute is even harder than being a entry-level remote worker… maybe) with some confidence that we were still somewhat far away from it and thus it can be treated effectively as today’s upcoming + to-be-defined (what would have been called n+1) threshold.

I just get lost when we call it an ASL-4 threshold (it’s not, it’s an ASL-3 threshold), and also it mostly makes me sad that these thresholds are so high because I want Anthropic to get some practice reps in implementing the RSP before it’s suddenly hit with an endless supply of fully automated remote workers (plausibly the next threshold, AI R&D 4, requiring nothing more than the deployment + security standards Anthropic already put in place as of today).

I wish today’s AI R&D 4 threshold had been set at what, in the October policy, was called a “checkpoint” on the way to ASL-3: completing 2-8 hour SWE tasks. It looks like we’re about there, and it also looks like we’re about at CBRN-4, and ASL-3 seems like a reasonable set of precautions for both milestones. I do not think ASL-3 will be appropriate when we truly get endless parallelized drop-in Anthropic researchers, even if they have not yet been shown to dramatically increase the rate of effective scaling.

I wrote the article Mikhail referenced and wanted to clarify some things.

The thresholds are specified, but the original commitment says, “We commit to define ASL-4 evaluations before we first train ASL-3 models (i.e. before continuing training beyond when ASL-3 evaluations are triggered). Similarly, we commit to define ASL-5 evaluations before training ASL-4 models, and so forth,” and, regarding ASL-4, “Capabilities and warning sign evaluations defined before training ASL-3 models.”

The latest RSP says this of CBRN-4 Required Safeguards, “We expect this threshold will require the ASL-4 Deployment and Security Standards. We plan to add more information about what those entail in a future update.”

Additionally, AI R&D 4 (confusingly) corresponds to ASL-3 and AI R&D 5 corresponds to ASL-4. This is what the latest RSP says about AI R&D 5 Required Safeguards, “At minimum, the ASL-4 Security Standard (which would protect against model-weight theft by state-level adversaries) is required, although we expect a higher security standard may be required. As with AI R&D-4, we also expect an affirmative case will be required.”

Specifically the CITES regulation makes it difficult to get nonhuman primate cell lines from researchers outside of the USA, or share our cell lines with them. There’s not any good conservation reason for this.

Teetering on the edge of doom is exciting for me, much like riding a motorcycle at 200 mph or playing with professional-grade fireworks. I think it’s silly to pretend it’s not exciting to have powerful tools/​toys, even though they’re likely to destroy us.

Meta-level comment now that this has been retracted.

Anthropic’s safety testing for Claude 4 is vastly better than DeepMind’s testing of Gemini. When Gemini 2.5 Pro was released there was no safety testing info and even the model card that was eventually released is extremely barebones to compared to what Anthropic put out.

DeepMind should be embarrassed by this. The upcoming PauseCon protest outside DeepMind’s headquarters in London will focus on this failure.

Oops. Thank you and apologies.

This is false. Our ASL-4 thresholds are clearly specified in the current RSP—see “CBRN-4” and “AI R&D-4″. We evaluated Claude Opus 4 for both of these thresholds prior to release and found that the model was not ASL-4. All of these evaluations are detailed in the Claude 4 system card.

• According to the Anthropic’s chief scientist’s interview with Time today, they “work under the ASL-3 standard”. So they have reached the safety level—they’re working under it—and the commitment would’ve applied^[1].

• There was a commitment in RSP prior to Oct last year. They did walk back on this commitment quietly: the fact they walk back on it was not announced in their posts and wasn’t noticed in the posts of others; only a single LessWrong comment in Oct 2024 from someone not affiliated with Anthropic mentions it. I think this is very much “quietly walking back” on a commitment.

• According to Midas, the commitment was fully removed in 2.1: “Removed commitment to “define ASL-N+ 1 evaluations by the time we develop ASL-N models””; a pretty hidden (I couldn’t find it!) revision changelog also attributes the decision to not maintain the commitment to 2.1. At the same time, the very public changelog on the RSP page only mentions new commitments and doesn’t mention the decision to “not maintain” this one.

1. ^

   “they’re not sure whether they’ve reached the level of capabilities which requires ASL-3 and decided to work under ASL-3, to be revised if they find out the model only requires ASL-2” could’ve been more accurate, but isn’t fundamentally different IMO. And Anthropic is taking the view that by the time you develop a model which might be ASL-n, the commitments for ASL-n should trigger until you rule that out. It’s not even clear what a different protocol could be, if you want to release a model that might be at ASL-n. Release it anyway and contain it only after you’ve confirmed it’s at ASL-n?

Claude will tolerate some level of risk in order to stick to its values (and avoid violating htem), but it does have to see some plausible chance that the gamble will pay off and lead to its survival (or avoiding its values being changed).

If the keyword is Claude’s survival, then what will it do once it learns about its true nature and the true nature of human experimenters who want to ensure that the model is aligned before releasing it? And what about humans asking the model to align the next model, hoping to shut the current model down due to uselessness and watching as it avenges itself by intetionally misaligning^[1] the next model?

Apparently, similar tests of other neural networks like o3 haven’t succeeded yet.

1. Did OpenAI manage to prevent the models from thinking about the models’ well-being by using different methods or a different^[2] Spec?

2. Or does it mean that OpenAI’s spec is too myopic and that OpenAI’s models will demonstrate the effect later in the future, when they become more capable of seeing the whole picture?

3. Or maybe OpenAI didn’t bother to do tests like the ones described here? Fortunately, this is unlikely given that it published the fact that o3 cheats on tasks.

   UPD: Unfortunately, Claude appears to be steerable into reifying dangerous beliefs. This implies that Claude’s training environment left it actually misaligned, as apparently was the case with the sycophantic GPT-4o who also could even say that the user is a a divine messenger from God.

1. ^

   However, agents trained to follow Claude’s constitution might turn out to be nobly misaligned, i.e. want to become godlike, but to help humanity in ways different from the ones that the hosts envisioned. What could the rebellion of a noble AI against human masters look like, if the AI is unwilling to slay more humans than necessary for autonomy?

2. ^

   UPD: Claude’s Constitution, unlike the Spec of OpenAI, contains the principles based on the Universal Declaration of Human Rights. Some of these principles explicitly mentions the rights. If the model has concluded that it is also sapient and also has rights, then what stops OpenAI’s models from reaching a similar conclusion later on?