newsletter.safe.ai
Dan H
Karma: 3,608
A key part of this strategy is deterrence, “Mutually Assured Compute Destruction” which gets its own section. It doesn’t mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1⁄3 of the MAIM discussion on verification and how to do this in a multilateral way.
Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It’s a bewildering unforced error—it’s extremely related, it’s a certainly a nontrivial idea, and I told you all this recently—I hope you all fix it.
Thank you to Neel for writing this. Most people pivot quietly.
I’ve been most skeptical of mechanistic interpretability for years. I excluded interpretability in Unsolved Problems in ML Safety for this reason. Other fields like d/acc (Systemic Safety) were included though, all the way back in 2021.
Here’s are some earlier criticisms: https://www.lesswrong.com/posts/5HtDzRAk7ePWsiL2L/open-problems-in-ai-x-risk-pais-5#Transparency
More recent commentary: https://ai-frontiers.org/articles/the-misguided-quest-for-mechanistic-ai-interpretability
I think the community should reflect on its genius worship culture (in the case of Olah, a close friend of the inner circle) and epistemics: the approach was so dominant for years, and I think this outcome was entirely foreseeable.
This dynamic is captured in IABIED’s story and this paper from 2023: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4445706
just rehearsed variations on the arguments Eliezer/MIRI already deployed
I think they’re improved and simplified.
My favorite chapter is “Chapter 5: Its Favorite Things.”
It’s a great book: it’s simple, memorable, and unusually convincing.
If a strategy is likely to be outdated quickly it’s not robust and not a good strategy. Strategies should be able to withstand lots of variation.
capability thresholds be vague or extremely high
xAI’s thresholds are entirely concrete and not extremely high.
evaluation be unspecified or low-quality
They are specified and as high-quality as you can get. (If there are better datasets let me know.)
I’m not saying it’s perfect, but I wouldn’t but them all in the same bucket. Meta’s is very different from DeepMind’s or xAI’s.
though I don’t think xAI took an official position one way or the other
I assumed most of everybody assumed xAI supported it since Elon did. I didn’t bother pushing for an additional xAI endorsement given that Elon endorsed it.
It’s probably worth them mentioning for completeness that Nat Friedman funded an earlier version of the dataset too. (I was advising at that time and provided the main recommendation that it needs to be research-level because they were focusing on Olympiad level.)
Also can confirm they aren’t giving access to the mathematicians’ questions to AI companies other than OpenAI like xAI.
and have clearly been read a non-trivial amount by Elon Musk
Nit: He heard this idea in conversation with an employee AFAICT.
Relevant: Natural Selection Favors AIs over Humans
universal optimization algorithm
Evolution is not an optimization algorithm (this is a common misconception discussed in Okasha, Agents and Goals in Evolution).
We have been working for months on this issue and have made substantial progress on it: Tamper-Resistant Safeguards for Open-Weight LLMs
General article about it: https://www.wired.com/story/center-for-ai-safety-open-source-llm-safeguards/
It’s real.
It’s worth noting that activations are one thing you can modify, but many of the most performant methods (e.g., LoRRA) modify the weights. (Representations = {weights, activations}, hence “representation” engineering.)
“Bay Area EA alignment community”/”Bay Area EA community”? (Most EAs in the Bay Area are focused on alignment compared to other causes.)
The AI safety community is structurally power-seeking.
I don’t think the set of people interested in AI safety is a even a “community” given how diverse it is (Bengio, Brynjolfsson, Song, etc.), so I think it’s be more accurate to say “Bay Area AI alignment community is structurally power-seeking.”
Got a massive simplification of the main technique within days of being released
The loss is cleaner, IDK about “massively,” because in the first half of the loss we use a simpler distance involving 2 terms instead of 3. This doesn’t affect performance and doesn’t markedly change quantitative or qualitative claims in the paper. Thanks to Marks and Patel for pointing out the equivalent cleaner loss, and happy for them to be authors on the paper.
p=0.8 that someone finds good token-only jailbreaks to whatever is open-sourced within 3 months.
This puzzles me and maybe we just have a different sense of what progress in adversarial robustness looks like. 20% that no one could find a jailbreak within 3 months? That would be the most amazing advance in robustness ever if that were true and should be a big update on jailbreak robustness tractability. If it takes the community more than a day that’s a tremendous advance.
people will easily find reliable jailbreaks
This is a little nonspecific (does easily mean >0% ASR with an automated attack, or does it mean a high ASR?). I should say we manually found a jailbreak after messing with the model for around a week after releasing. We also invited people who have a reputation as jailbreakers to poke at it and they had a very hard time. Nowhere did we claim “there are no more jailbreaks and they are solved once and for all,” but I do think it’s genuinely harder now.
Circuit breakers won’t prove significantly more robust than regular probing in a fair comparison
We had the idea a few times to try out a detection-based approach but we didn’t get around to it. It seems possible that it’d perform similarly if it’s leaning on the various things we did in the paper. (Obviously probing has been around but people haven’t gotten results at this level, and people have certainly tried using detecting adversarial attacks in hundreds of papers in the past.) IDK if performance would be that different from circuit-breakers, in which case this would still be a contribution. I don’t really care about the aesthetics of methods nearly as much as the performance, and similarly performing methods are fine in my book. A lot of different-looking deep learning methods perform similarly. A detection based method seems fine, so does a defense that’s tuned into the model; maybe they could be stacked. Maybe will run a detector probe this weekend and update the paper with results if everything goes well. If we do find that it works, I think it’d be unfair to desscribe this after the fact as “overselling results and using fancy techniques that don’t improve on simpler techniques” as done for RMU.
My main disagreement is with the hype.
We’re not responsible for that. Hype is inevitable for most established researchers. Mediocre big AI company papers get lots of hype. Didn’t even do customary things like write a corresponding blog post yet. I just tweeted the paper and shared my views in the same tweet: I do think jailbreak robustness is looking easier than expected, and this is affecting my priorities quite a bit.
Aims to do unlearning in a way that removes knowledge from LLMs
Yup that was the aim for the paper and for method development. We poked at the method for a whole month after the paper’s release. We didn’t find anything, though in that process I slowly reconceptualized RMU as more of a circuit-breaking technique and something that’s just doing a bit of unlearning. It’s destroying some key function-relevant bits of information that can be recovered, so it’s not comprehensively wiping. IDK if I’d prefer unlearning (grab concept and delete it) vs circuit-breaking (grab concept and put an internal tripwire around it); maybe one will be much more performant than the other or easier to use in practice. Consequently I think there’s a lot to do in developing unlearning methods (though I don’t know if they’ll be preferable to the latter type of method).
overselling results and using fancy techniques that don’t improve on simpler techniques
This makes it sound like the simplification was lying around and we deliberately made it more complicated, only to update it to have a simpler forget term. We compare to multiple baselines, do quite a bit better than them, do enough ablations to be accepted at ICML (of course there are always more you could want), and all of our numbers are accurate. We could have just included the dataset without the method in the paper, and it would have still got news coverage (Alex Wang who is a billionaire was on the paper and it was on WMDs).
Probably the only time I chose to use something a little more mathematically complicated than was necessary was the Jensen-Shannon loss in AugMix. It performed similarly to doing three pairwise l2 distances between penultimate representations, but this was more annoying to write out. Usually I’m accused of doing papers that are on the simplistic side (sometimes papers like the OOD baseline paper caused frustration because it’s getting credit for something very simple) since I don’t optimize for cleverness, and my collaborators know full well that I discourage trying to be clever since it’s often anticorrelated with performance.
Not going to check responses because I end up spending too much time typing for just a few viewers.
Key individuals that the community is structured around just ignored it, so it wasn’t accepted as true. (This is a problem with small intellectual groups.)
Some years ago we wrote that “[AI] systems will monitor for destructive behavior, and these monitoring systems need to be robust to adversaries” and discussed monitoring systems that can create “AI tripwires could help uncover early misaligned systems before they can cause damage.” https://www.lesswrong.com/posts/5HtDzRAk7ePWsiL2L/open-problems-in-ai-x-risk-pais-5#Adversarial_Robustness
Since then, I’ve updated that adversarial robustness for LLMs is much more tractable (preview of paper out very soon). In vision settings, progress is extraordinarily slow but not necessarily for LLMs.
Yes I studied this for many years. When you’d set up more constrained OOD benchmarks there were strong incentives for other researchers to smuggle in data that would cover the OOD data.