newsletter.safe.ai
Dan H
Some years ago we wrote that “[AI] systems will monitor for destructive behavior, and these monitoring systems need to be robust to adversaries” and discussed monitoring systems that can create “AI tripwires could help uncover early misaligned systems before they can cause damage.” https://www.lesswrong.com/posts/5HtDzRAk7ePWsiL2L/open-problems-in-ai-x-risk-pais-5#Adversarial_Robustness
Since then, I’ve updated that adversarial robustness for LLMs is much more tractable (preview of paper out very soon). In vision settings, progress is extraordinarily slow but not necessarily for LLMs.
Various comments:
I wouldn’t call this “AI lab watch.” “Lab” has the connotation that these are small projects instead of multibillion dollar corporate behemoths.
“deployment” initially sounds like “are they using output filters which harm UX in deployment”, but instead this seems to be penalizing organizations if they open source. This seems odd since open sourcing is not clearly bad right now. The description also makes claims like “Meta release all of their weights”—they don’t release many image/video models because of deepfakes, so they are doing some cost-benefit analysis. Zuck: “So we want to see what other people are observing, what we’re observing, what we can mitigate, and then we’ll make our assessment on whether we can make it open source.” If this is mainly a penalty against open sourcing the label should be clearer.
“Commit to do pre-deployment risk assessment” They’ve all committed to this in the WH voluntary commitments and I think the labs are doing things on this front.
“Do risk assessment” These companies have signed on to WH voluntary commitments so are all checking for these things, and the EO says to check for these hazards too. This is why it’s surprising to see Microsoft have 1% given that they’re all checking for these hazards.
Looking at the scoring criteria, this seems highly fixated on rogue AIs, but I understand I’m saying that to the original forum of these concerns. Risk assessment’s scoring doesn’t really seem to prioritize bio x-risk as much as scheming AIs. This is strange because if we’re focused on rogue AIs I’d put a half the priority of risk mitigation while the model is training. Many rogue AI people may think half of the time the AI will kill everyone is when the model is “training” (because it will escape during that time).
The first sentence of this site says the focus is on “extreme risks” but it seems the focus is mainly on rogue AIs. This should be upfront that this is from the perspective that loss of control is the main extreme risk, rather than positioning itself as a comprehensive safety tracker. If I were tracking rogue AI risks, I’d probably drill down to what they plan to do with automated AI R&D/intelligence explosions.
“Training” This seems to give way more weight to rogue AI stuff. Red teaming is actually assessable, but instead you’re giving twice the points to if they have someone “work on scalable oversight.” This seems like an EA vibes check rather than actually measuring something. This also seems like triple counting since it’s highly associated with the “scalable alignment” section and the “alignment program” section. This doesn’t even require that they use the technique for the big models they train and deploy. Independently, capabilities work related to building superintelligences can easily be framed as scalable oversight, so this doesn’t set good incentives. Separately, at the end this also gives lots of points for voluntary (read: easily breakable) commitments. These should not be trusted and I think the amount of lipservice points is odd.
“Security” As I said on EAF the security scores are suspicious to me and even look backward. The major tech companies have much more experience protecting assets (e.g., clouds need to be highly secure) than startups like Anthropic and OpenAI. It takes years building up robust information security and the older companies have a sizable advantage.
“internal governance” scores seem odd. Older, larger institutions such as Microsoft and Google have many constraints and processes and don’t have leaders who can unilaterally make decisions as easily, compared to startups. Their CEOs are also more fireable (OpenAI), and their board members aren’t all selected by the founder (Anthropic). This seems highly keyed into if they are just a PBC or non-profit. In practice PBC just makes it harder to sue, but Zuck has such control of his company that getting successfully sued for not upholding his fiduciary duty to shareholders seems unlikely. It seems 20% of the points is not using non-disparagement agreements?? 30% is for whistleblower policies; CA has many whistleblower protections if I recall correctly. No points for a chief risk officer or internal audit committee?
“Alignment program” “Other labs near the frontier publish basically no alignment research” Meta publishes dozens of papers they call “alignment”; these actually don’t feel that dissimilar to papers like Constitutional AI-like papers (https://twitter.com/jaseweston/status/1748158323369611577 https://twitter.com/jaseweston/status/1770626660338913666 https://arxiv.org/pdf/2305.11206 ). These papers aren’t posted to LW but they definitely exist. To be clear I think this is general capabilities but this community seems to think differently. Alignment cannot be “did it come from EA authors” and it probably should not be “does it use alignment in its title.” You’ll need to be clear how this distinction is drawn.
Meta has people working on safety and CBRN+cyber + adversarial robustness etc. I think they’re doing a good job (here are two papers from the last month: https://arxiv.org/pdf/2404.13161v1 https://arxiv.org/pdf/2404.16873).
As is, I think this is a little too quirky and not ecumenical enough for it to generate social pressure.
There should be points for how the organizations act wrt to legislation. In the SB 1047 bill that CAIS co-sponsored, we’ve noticed some AI companies to be much more antagonistic than others. I think is is probably a larger differentiator for an organization’s goodness or badness.
(Won’t read replies since I have a lot to do today.)
is novel compared to… RepE
This is inaccurate, and I suggest reading our paper: https://arxiv.org/abs/2310.01405
Demonstrate full ablation of the refusal behavior with much less effect on coherence
In our paper and notebook we show the models are coherent.
Investigate projection
We did investigate projection too (we use it for concept removal in the RepE paper) but didn’t find a substantial benefit for jailbreaking.
harmful/harmless instructions
We use harmful/harmless instructions.
Find that projecting away the (same, linear) feature at all layers improves upon steering at a single layer
In the RepE paper we target multiple layers as well.
Test on many different models
The paper used Vicuna, the notebook used Llama 2. Throughout the paper we showed the general approach worked on many different models.
Describe a way of turning this into a weight-edit
We do weight editing in the RepE paper (that’s why it’s called RepE instead of ActE).
but generally people should be free to post research updates on LW/AF that don’t have a complete thorough lit review / related work section.
I agree if they simultaneously agree that they don’t expect the post to be cited. These can’t posture themselves as academic artifacts (“Citing this work” indicates that’s the expectation) and fail to mention related work. I don’t think you should expect people to treat it as related work if you don’t cover related work yourself.
Otherwise there’s a race to the bottom and it makes sense to post daily research notes and flag plant that way. This increases pressure on researchers further.
including refusal-bypassing-related ones
The prior work that is covered in the document is generally less related (fine-tuning removal of safeguards, truth directions) compared to these directly relevant ones. This is an unusual citation pattern and gives the impression that the artifact is making more progress/advancing understanding than it actually is.
I’ll note pretty much every time I mention something isn’t following academic standards on LW I get ganged up on and I find it pretty weird. I’ve reviewed, organized, and can be senior area chair at ML conferences and know the standards well. Perhaps this response is consistent because it feels like an outside community imposing things on LW.
From Andy Zou:
Thank you for your reply.
Model interventions to bypass refusal are not discussed in Section 6.2.
We perform model interventions to robustify refusal (your section on “Adding in the “refusal direction” to induce refusal”). Bypassing refusal, which we do in the GitHub demo, is merely adding a negative sign to the direction. Either of these experiments show refusal can be mediated by a single direction, in keeping with the title of this post.
we examined Section 6.2 carefully before writing our work
Not mentioning it anywhere in your work is highly unusual given its extreme similarity. Knowingly not citing probably the most related experiments is generally considered plagiarism or citation misconduct, though this is a blog post so norms for thoroughness are weaker. (lightly edited by Dan for clarity)
Ablating vs. Addition
We perform a linear combination operation on the representation. Projecting out the direction is one instantiation of it with a particular coefficient, which is not necessary as shown by our GitHub demo. (Dan: we experimented with projection in the RepE paper and didn’t find it was worth the complication. We look forward to any results suggesting a strong improvement.)
--
Please reach out to Andy if you want to talk more about this.
Edit: The work is prior art (it’s been over six months+standard accessible format), the PIs are aware of the work (the PI of this work has spoken about it with Dan months ago, and the lead author spoke with Andy about the paper months ago), and its relative similarity is probably higher than any other artifact. When this is on arXiv we’re asking you to cite the related work and acknowledge its similarities rather than acting like these have little to do with each other/not mentioning it. Retaliating by some people dogpile voting/ganging up on this comment to bury sloppy behavior/an embarrassing oversight is not the right response (went to −18 very quickly).
Edit 2: On X, Neel “agree[s] it’s highly relevant” and that he’ll cite it. Assuming it’s covered fairly and reasonably, this resolves the situation.
Edit 3: I think not citing it isn’t a big deal because I think of LW as a place for ml research rough drafts, in which errors will happen. But if some are thinking it’s at the level of an academic artifact/is citable content/is an expectation others cite it going forward, then failing to mention extremely similar results would actually be a bigger deal. Currently I’ll think it’s the former.
From Andy Zou:
Section 6.2 of the Representation Engineering paper shows exactly this (video). There is also a demo here in the paper’s repository which shows that adding a “harmlessness” direction to a model’s representation can effectively jailbreak the model.
Going further, we show that using a piece-wise linear operator can further boost model robustness to jailbreaks while limiting exaggerated refusal. This should be cited.
If people are interested, many of these concepts and others are discussed in the context of AI safety in this publicly available chapter: https://www.aisafetybook.com/textbook/4-1
Here is a chapter from an upcoming textbook on complex systems with discussion of their application to AI safety: https://www.aisafetybook.com/textbook/5-1
> My understanding is that we already know that backdoors are hard to remove.
We don’t actually find that backdoors are always hard to remove!We did already know that backdoors often (from the title) “Persist Through Safety Training.” This phenomenon studied here and elsewhere is being taken as the main update in favor of AI x-risk. This doesn’t establish probability of the hazard, but it reminds us that backdoor hazards can persist if present.
I think it’s very easy to argue the hazard could emerge from malicious actors poisoning pretraining data, and harder to argue it would arise naturally. AI security researchers such as Carlini et al. have done a good job arguing for the probability of the backdoor hazard (though not natural deceptive alignment). (I think malicious actors unleashing rogue AIs is a concern for the reasons bio GCRs are a concern; if one does it, it could be devastating.)
I think this paper shows the community at large will pay orders of magnitude more attention to a research area when there is, in @TurnTrout’s words, AGI threat scenario “window dressing,” or when players from an EA-coded group research a topic. (I’ve been suggesting more attention to backdoors since maybe 2019; here’s a video from a few years ago about the topic; we’ve also run competitions at NeurIPS with thousands of submissions on backdoors.) Ideally the community would pay more attention to relevant research microcosms that don’t have the window dressing.
I think AI security-related topics have a very good track record of being relevant for x-risk (backdoors, unlearning, adversarial robustness). It’s a been better portfolio than the EA AGI x-risk community portfolio (decision theory, feature visualizations, inverse reinforcement learning, natural abstractions, infrabayesianism, etc.). At a high level its saying power is because AI security is largely about extreme reliability; extreme reliability is not automatically provided by scaling, but most other desiderata are (e.g., commonsense understanding of what people like and dislike).
A request: Could Anthropic employees not call supervised fine-tuning and related techniques “safety training?” OpenAI/Anthropic have made “alignment” in the ML community become synonymous with fine-tuning, which is a big loss. Calling this “alignment training” consistently would help reduce the watering down of the word “safety.”
I agree that this is an important frontier (and am doing a big project on this).
Almost all datasets have label noise. Most 4-way multiple choice NLP datasets collected with MTurk have ~10% label noise, very roughly. My guess is MMLU has 1-2%. I’ve seen these sorts of label noise posts/papers/videos come out for pretty much every major dataset (CIFAR, ImageNet, etc.).
The purpose of this is to test and forecast problem-solving ability, using examples that substantially lose informativeness in the presence of Python executable scripts. I think this restriction isn’t an ideological statement about what sort of alignment strategies we want.
I think there’s a clear enough distinction between Transformers with and without tools. The human brain can also be viewed as a computational machine, but when exams say “no calculators,” they’re not banning mental calculation, rather specific tools.
It was specified in the beginning of 2022 in https://www.metaculus.com/questions/8840/ai-performance-on-math-dataset-before-2025/#comment-77113 In your metaculus question you may not have added that restriction. I think the question is much less interesting/informative if it does not have that restriction. The questions were designed assuming there’s no calculator access. It’s well-known many AIME problems are dramatically easier with a powerful calculator, since one could bash 1000 options and find the number that works for many problems. That’s no longer testing problem-solving ability; it tests the ability to set up a simple script so loses nearly all the signal. Separately, the human results we collected was with a no calculator restriction. AMC/AIME exams have a no calculator restriction. There are different maths competitions that allow calculators, but there are substantially fewer quality questions of that sort.
I think MMLU+calculator is fine though since many of the exams from which MMLU draws allow calculators.
Usage of calculators and scripts are disqualifying on many competitive maths exams. Results obtained this way wouldn’t count (this was specified some years back). However, that is an interesting paper worth checking out.
Neurotechnology, brain computer interface, whole brain emulation, and “lo-fi” uploading approaches to produce human-aligned software intelligence
Thank you for doing this.
There’s a literature on this topic. (paper list, lecture/slides/homework)
Plug: CAIS is funding constrained.
Why was the AI Alignment community so unprepared for engaging with the wider world when the moment finally came?
In 2022, I think it was becoming clear that there’d be a huge flood of interest. Why did I think this? Here are some reasons: I’ve long thought that once MMLU performance crosses a threshold, Google would start to view AI as an existential threat to their search engine, and it seemed like in 2023 that threshold would be crossed. Second, at a rich person’s party, there were many highly plugged-in elites who were starting to get much more anxious about AI (this was before ChatGPT), which updated me that the tide may turn soon.
Since I believed the interest would shift so much, I changed how I spent my time a lot in 2022: I started doing substantially less technical work to instead work on outreach and orienting documents. Here are several projects I did, some for targeted for the expert community and some targeted towards the general public:
We ran an AI arguments writing competition. After seeing that we could not crowdsource AI risk writing to the community through contests last year, I also started work on An Overview of Catastrophic Risks last winter. We had a viable draft several in April, but then I decided to restructure it, which required rewriting it and making it longer. This document was partly a synthesis of the submissions from the first round of the AI arguments competition, so fortunately the competition did not go to waste. Apologies the document took so long.
Last summer and fall, I worked on explaining a different AI risk to a lay audience in Natural Selection Favors AIs over Humans (apparently this doom path polls much better than treacherous turn stories; I held onto the finished paper for months and waited for GPT-4′s release before releasing it to have good timing).
X-Risk Analysis for AI Research tries to systematically articulate how to analyze AI research’s relation to x-risk for a technical audience. It was my first go at writing about AI x-risk for the ML research community. I recognize this paper was around a year ahead of its time and maybe I should have held onto it to release it later.
Finally, after a conversation with Kelsey Piper and the aforementioned party, I was inspired to work on a textbook An Introduction to AI Safety, Ethics, and Society. This is by far the largest writing project I’ve been a part of. Currently, the only way to become an AI x-risk expert is to live in Berkeley. I want to reduce this barrier as much as possible, relate AI risk to existing literatures, and let people have a more holistic understanding of AI risk (I think people should have a basic understanding of all of corrigibility, international coordination for AI, deception, etc.). This book is not an ML PhD topics book; it’s more to give generalists good models. The textbook’s contents will start to be released section-by-section on a daily basis starting late this month or next month. Normally textbooks take several years to make, so I’m happy this will be out relatively quickly.
One project we only started in 2023 is newsletter, so we can’t claim prescience for that.
If you want more AI risk outputs, CAIS is funding-constrained and is currently fundraising for a writer.
Key individuals that the community is structured around just ignored it, so it wasn’t accepted as true. (This is a problem with small intellectual groups.)