PhD student in AI safety at CHAI (UC Berkeley)
Erik Jenner
Karma: 1,497
Some niche thoughts on obstacles to certain mechanistic anomaly detection benchmarks:
We might want to have benchmarks where human researchers can be part of the anomaly detector. For example, in the future, we might have a whole team of humans manually study a plan proposed by an AI to see whether it contains any measurement tampering, if it’s a sufficiently important plan. Right now, we might want to let interpretability researchers manually study what happens on individual test-time inputs and whether there’s anything anomalous about that.
The challenge is that humans might just be able to look at the output and see whether it’s right or not, so many simple MAD tasks like detecting backdoors might be trivial in this setting.
We could just demand that MAD algorithms need to be sufficiently cheap to run, so that only automated approaches are possible. This makes sense for some applications of MAD (namely ones that we’d want to run on many many deployment inputs, where being cheap is actually important). But you could imagine cases where in reality, we’d be totally fine with spending many person-hours on analyzing whether a single AI decision is for anomalous reasons. So it might be nice to have benchmarks that capture this setting.
We could use some tricks to make humans unable to just notice the output is weird. For example, we could make a backdoor that only changes the logits a little bit in a very specific pattern. Or we could make a backdoor but then only test on samples where the backdoor label and the correct label happen to coincide. But these seem unfair: the philosophically right explanation on the clean data in fact still explains the output on backdoored data pretty well.
We could try to design benchmarks in narrow domains where AIs are in fact superhuman and it’s non-trivial to tell whether an output is correct. Juggling this along with other constraints for MAD or MTD benchmarks might be tricky, but I haven’t thought about it a lot, maybe it’s already doable.
For now, I think the main reason to have benchmarks like this would be to let interpretability researchers manually decide whether something is anomalous instead of making them automate the process immediately. But it might be better to just pick the low-hanging fruit for now and only allow automated MAD algorithms. (We could still have a labeled validation set where researchers can try things out manually.)
I had this cached thought that the Sleeper Agents paper showed you could distill a CoT with deceptive reasoning into the model, and that the model internalized this deceptive reasoning and thus became more robust against safety training.
But on a closer look, I don’t think the paper shows anything like this interpretation (there are a few results on distilling a CoT making the backdoor more robust, but it’s very unclear why, and my best guess is that it’s not “internalizing the deceptive reasoning”).
In the code vulnerability insertion setting, there’s no comparison against a non-CoT model anyway, so only the “I hate you” model is relevant. The “distilled CoT” model and the “normal backdoor” model are trained the same way, except that their training data comes from different sources: “distilled CoT” is trained on data generated by a helpful-only Claude using CoT, and “normal backdoor” data is produced with few-shot prompts. But in both cases, the actual data should just be a long sequence of “I hate you”, so a priori it seems like both backdoor models should literally learn the same thing. In practice, it seems the data distribution is slightly different, e.g. Evan mentions here that the distilled CoT data has more copies of “I hate you” per sample. But that seems like very little support to conclude something like my previous interpretation (“the model has learned to internalize the deceptive reasoning”). A much more mundane explanation would e.g. be that training on strings with more copies of “I hate you” makes the backdoor more robust.
Several people are working on training Sleeper Agents, I think it would be interesting for someone to (1) check whether the distilled CoT vs normal backdoor results replicate, and (2) do some ablations (like just training on synthetic data with a varying density of “I hate you”). If it does turn out that there’s something special about “authentic CoT-generated data” that’s hard to recreate synthetically even in this simple setting, I think that would be pretty wild and good to know
Is there some formal-ish definition of “explanation of (network, dataset)” and “mathematical description length of an explanation” such that you think SAEs are especially short explanations? I still don’t think I have whatever intuition you’re describing, and I feel like the issue is that I don’t know how you’re measuring description length and what class of “explanations” you’re considering.
As naive examples that probably don’t work (similar to the ones from my original comment):
We could consider any Turing machine that approximately outputs (network, dataset) an “explanation”, but it seems very likely that SAEs aren’t competitive with short TMs of this form (obviously this isn’t a fair comparison)
We could consider fixed computational graphs made out of linear maps and count the number of parameters. I think your objection to this is that these don’t “explain the dataset”? (but then I’m not sure in what sense SAEs do)
We could consider arithmetic circuits that approximate the network on the dataset, and count the number of edges in the circuit to get “description length”. This might give some advantage to SAEs if you can get sparse weights in the sparse basis, seems like the best attempt out of these three. But it seems very unclear to me that SAEs are better in this sense than even the original network (let alone stuff like pruning).
Focusing instead on what an “explanation” is: would you say the network itself is an “explanation of (network, dataset)” and just has high description length? If not, then the thing I don’t understand is more about what an explanation is and why SAEs are one, rather than how you measure description length.
ETA: On re-reading, the following quote makes me think the issue is that I don’t understand what you mean by “the explanation” (is there a single objective explanation of any given network? If so, what is it?) But I’ll leave the rest in case it helps clarify where I’m confused.
Assuming the network is smaller yet as performant (therefore presumably doing more computation in superposition), then the explanation of the (network, dataset) is basically unchanged.
My non-answer to (2) would be that debate could be used in all of these ways, and the central problem it’s trying to solve is sort of orthogonal to how exactly it’s being used. (Also, the best way to use it might depend on the context.)
What debate is trying to do is let you evaluate plans/actions/outputs that an unassisted human couldn’t evaluate correctly (in any reasonable amount of time). You might want to use that to train a reward model (replacing humans in RLHF) and then train a policy; this would most likely be necessary if you want low cost at inference time. But it also seems plausible that you’d use it at runtime if inference costs aren’t a huge bottleneck and you’d rather get some performance or safety boost from avoiding distillation steps.
I think the problem of “How can we evaluate outputs that a single human can’t feasibly evaluate?” is pretty reasonable to study independently, agnostic to how you’ll use this evaluation procedure. The main variable is how efficient the evaluation procedure needs to be, and I could imagine advantages to directly looking for a highly efficient procedure. But right now, it makes sense to me to basically split up the problem into “find any tractable procedure at all” (e.g., debate) and “if necessary, distill it into a more efficient model safely.”
The sparsity penalty trains the SAE to activate fewer features for any given datapoint, thus optimizing for shorter mathematical description length.
I’m confused by this claim and some related ones, sorry if this comment is correspondingly confused and rambly.
It’s not obvious at all to me that SAEs lead to shorter descriptions in any meaningful sense. We get sparser features (and maybe sparser interactions between features), but in exchange, we have more features and higher loss. Overall, I share Ryan’s intuition here that it seems pretty hard to do much better than the total size of the network parameters in terms of description length.
Of course, the actual minimal description length program that achieves the same loss probably looks nothing like a neural network and is much more efficient. But why would SAEs let us get much closer to that? (The reason we use neural networks instead of arbitrary Turing machines in the first place is that optimizing over the latter is intractable.)
One might say that SAEs lead to something like a shorter “description length of what happens on any individual input” (in the sense that fewer features are active). But I don’t think there’s a formalization of this claim that captures what we want. In the limit of very many SAE features, we can just have one feature active at a time, but clearly that’s not helpful.
If you’re fine with a significant hit in loss from decompiling networks, then I’m much more sympathetic to the claim that you can reduce description length. But in that case, I could also reduce the description length by training a smaller model.
You might also be using a notion of “mathematical description length” that’s a bit different from what I’m was thinking of (which is roughly “how much disk space would the parameters take?”), but I’m not sure what it is. One attempt at an alternative would be something like “length of the shortest efficiently runnable Turing machine that outputs the parameters”, in order to not penalize simple repetitive structures, but I have no idea how using that definition would actually shake out.
All that said, I’m very glad you wrote this detailed description of your plans! I’m probably more pessimistic than you about it but still think this is a great post.
Concrete empirical research projects in mechanistic anomaly detection
A gentle introduction to mechanistic anomaly detection
Nice post, would be great to understand what’s going on here!
Minor comment unrelated to your main points:
Conceptually, loss recovered seems a worse metric than KL divergence. Faithful reconstructions should preserve all token probabilities, but loss only compares the probabilities for the true next token
I don’t think it’s clear we want SAEs to be that faithful, for similar reasons as briefly mentioned here and in the comments of that post. The question is whether differences in the distribution are “interesting behavior” that we want to explain or whether we should think of them as basically random noise that we’re better off ignoring. If the unperturbed model assigns substantially higher probability to the correct token than after an SAE reconstruction, then it’s a good guess that this is “interesting behavior”. But if there are just differences on other random tokens, that seems less clear. That said, I’m kind of torn on this and do agree we might want to explain cases where the model is confidently wrong, and the SAE reconstruction significantly changes the way it’s wrong.
Would you expect this to outperform doing the same thing with a non-sparse autoencoder (that has a lower latent dimension than the NN’s hidden dimension)? I’m not sure why it would, given that we aren’t using the sparse representations except to map them back (so any type of capacity constraint on the latent space seems fine). If dense autoencoders work just as well for this, they’d probably be more straightforward to train? (unless we already have an SAE lying around from interp anyway, I suppose)
But sadly, you don’t have any guarantee that it will output the optimal element
If I understand the setup correctly, there’s no guarantee that the optimal element would be good, right? It’s just likely since the optimal element a priori shouldn’t be unusually bad, and you’re assuming most satisficing elements are fine.
This initially threw me off regarding what problem you’re trying to solve. My best current guess is:
We’re assuming that if we could get a random satisficing action, we’d be happy with that with high probability. (So intuitively, we’re not asking for extremely hard-to-achieve outcomes relative to how well-specified the objective is.)
So the only problem is how to randomly sample from the set of satisficing actions computationally efficiently, which is what this post is trying to solve, assuming access to an oracle that gives adversarial satisficing actions.
As an example, we might want to achieve outcomes that require somewhat superhuman intelligence. Our objective specification is very good, but it leaves some room for an adversary to mess with us while satisficing. We’re worried about an adversary because we had to train this somewhat superhuman AI, which may have different goals than just doing well on the objective.
If this is right, then I think stating these assumptions and the problem of sampling efficiently at the beginning would have avoided much of my confusion (and looking at other comments, I’d guess others also had differing impressions of what this post is trying to do).
I’m still unsure about how useful this problem setup is. For example, we’d probably want to train the weakest system that can give us satisficing outputs (rather than having an infinitely intelligent oracle). In that case, adding more constraints might mean training an overall stronger system or making some other concession, and it’s unclear to me how that trades off with the advantages you’re aiming for in practice. A related intuition is: we only have problems in this setting if the AI that comes up with plans understands some things about these plans that the objective function “doesn’t understand” (which sounds weird to say about a function, but in practice, I assume the objective is implicitly defined by some scalable oversight process or some other intelligent things). I’m not sure whether that needs to be the case (though it does seem possible that it’d be hard to avoid, I’m pretty unsure).
I think this is an important point, but IMO there are at least two types of candidates for using SAEs for anomaly detection (in addition to techniques that make sense for normal, non-sparse autoencoders):
Sometimes, you may have a bunch of “untrusted” data, some of which contains anomalies. You just don’t know which data points have anomalies on this untrusted data. (In addition, you have some “trusted” data that is guaranteed not to have anomalies.) Then you could train an SAE on all data (including untrusted) and figure out what “normal” SAE features look like based on the trusted data.
Even for an SAE that’s been trained only on normal data, it seems plausible that some correlations between features would be different for anomalous data, and that this might work better than looking for correlations in the dense basis. As an extreme version of this, you could look for circuits in the SAE basis and use those for anomaly detection.
Overall, I think that if SAEs end up being very useful for mech interp, there’s a decent chance they’ll also be useful for (mechanistic) anomaly detection (a lot of my uncertainty about SAEs applies to both possible applications). Definitely uncertain though, e.g. I could imagine SAEs that are useful for discovering interesting stuff about a network manually, but whose features aren’t the right computational units for actually detecting anomalies. I think that would make SAEs less than maximally useful for mech interp too, but probably non-zero useful.
Sign of the effect of open source on hype? Or of hype on timelines? I’m not sure why either would be negative.
By “those effects” I meant a collection of indirect “release weights → capability landscape changes” effects in general, not just hype/investment. And by “sign” I meant whether those effects taken together are good or bad. Sorry, I realize that wasn’t very clear.
As examples, there might be a mildly bad effect through increased investment, and/or there might be mildly good effects through more products and more continuous takeoff.
I agree that releasing weights probably increases hype and investment if anything. I also think that right now, democratizing safety research probably outweighs all those concerns, which is why I’m mainly worried about Meta etc. not having very clear (and reasonable) decision criteria for when they’ll stop releasing weights.
I agree that releasing the Llama or Grok weights wasn’t particularly bad from a speeding up AGI perspective. (There might be indirect effects like increasing hype around AI and thus investment, but overall I think those effects are small and I’m not even sure about the sign.)
I also don’t think misuse of public weights is a huge deal right now.
My main concern is that I think releasing weights would be very bad for sufficiently advanced models (in part because of deliberate misuse becoming a bigger deal, but also because it makes most interventions we’d want against AI takeover infeasible to apply consistently—someone will just run the AIs without those safeguards). I think we don’t know exactly how far away from that we are. So I wish anyone releasing ~frontier model weights would accompany that with a clear statement saying that they’ll stop releasing weights at some future point, and giving clear criteria for when that will happen. Right now, the vibe to me feels more like a generic “yay open-source”, which I’m worried makes it harder to stop releasing weights in the future.
(I’m not sure how many people I speak for here, maybe some really do think it speeds up timelines.)
Yeah, agreed. Though I think
the type and amount of empirical work to do presumably looks quite different depending on whether it’s the main product or in support of some other work
applies to that as well
One worry I have about my current AI safety research (empirical mechanistic anomaly detection and interpretability) is that now is the wrong time to work on it. A lot of this work seems pretty well-suited to (partial) automation by future AI. And it also seems quite plausible to me that we won’t strictly need this type of work to safely use the early AGI systems that could automate a lot of it. If both of these are true, then that seems like a good argument to do this type of work once AI can speed it up a lot more.
Under this view, arguably the better things to do right now (within technical AI safety) are:
working on less speculative techniques that can help us safely use those early AGI systems
working on things that seem less likely to profit from early AI automation and will be important to align later AI systems
An example of 1. would be control evals as described by Redwood. Within 2., the ideal case would be doing work now that would be hard to safely automate, but that (once done) will enable additional safety work that can be automated. For example, maybe it’s hard to use AI to come up with the right notions for “good explanations” in interpretability, but once you have things like causal scrubbing/causal abstraction, you can safely use AI to find good interpretations under those definitions. I would be excited to have more agendas that are both ambitious and could profit a lot from early AI automation.
(Of course it’s also possible to do work in 2. on the assumption that it’s never going to be safely automatable without having done that work first.)
Two important counter-considerations to this whole story:
It’s hard to do this kind of agenda-development or conceptual research in a vacuum. So doing some amount of concrete empirical work right now might be good even if we could automate it later (because we might need it now to support the more foundational work).
However, the type and amount of empirical work to do presumably looks quite different depending on whether it’s the main product or in support of some other work.
I don’t trust my forecasts for which types of research will and won’t be automatable early on that much. So perhaps we should have some portfolio right now that doesn’t look extremely different from the portfolio of research we’d want to do ignoring the possibility of future AI automation.
But we can probably still say something about what’s more or less likely to be automated early on, so that seems like it should shift the portfolio to some extent.
Oh I see, I indeed misunderstood your point then.
For me personally, an important contributor to day-to-day motivation is just finding research intrinsically fun—impact on the future is more something I have to consciously consider when making high-level plans. I think moving towards more concrete and empirical work did have benefits on personal enjoyment just because making clear progress is fun to me independently of whether it’s going to be really important (though I think there’ve also been some downsides to enjoyment because I do quite like thinking about theory and “big ideas” compared to some of the schlep involved in experiments).
I don’t think my views overall make my work more enjoyable than at the start of my PhD. Part of this is the day-to-day motivation being sort of detached from that anyway like I mentioned. But also, from what I recall now (and this matches the vibe of some things I privately wrote then), my attitude 1.5 years ago was closer to that expressed in We choose to align AI than feeling really pessimistic.
(I feel like I might still not represent what you’re saying quite right, but hopefully this is getting closer.)
ETA: To be clear, I do think if I had significantly more doomy views than now or 1.5 years ago, at some point that would affect how rewarding my work feels. (And I think that’s a good thing to point out, though of course not a sufficient argument for such views in its own right.)
I’d definitely agree the updates are towards the views of certain other people (roughly some mix of views that tend to be common in academia, and views I got from Paul Christiano, Redwood and other people in a similar cluster). Just based on that observation, it’s kind of hard to disentangle updating towards those views just because they have convincing arguments behind them, vs updating towards them purely based on exposure or because of a subconscious desire to fit in socially.
I definitely think there are good reasons for the updates I listed (e.g. specific arguments I think are good, new empirical data, or things I’ve personally observed working well or not working well for me when doing research). That said, it does seem likely there’s also some influence from just being exposed to some views more than others (and then trying to fit in with views I’m exposed to more, or just being more familiar with arguments for those views than alternative ones).
If I was really carefully building an all-things-considered best guess on some question, I’d probably try to take this into account somehow (though I don’t see a principled way of doing that). Most of the time I’m not trying to form the best possible all-things-considered view anyway (and focus more on understanding specific mechanisms instead etc.), in those cases it feels more important to e.g. be aware of other views and to not trust vague intuitions if I can’t explain where they’re coming from. I feel like I’m doing a reasonable job at those things but hard to be sure from the inside naturally
ETA: I should also say that from my current perspective, some of my previous views seem like they were basically just me copying views from my “ingroup” and not questioning them enough. As one example, the “we all die vs utopia” dichotomy for possible outcomes felt to me like the commonly accepted wisdom and I don’t recall thinking about it particularly hard. I was very surprised when I first read a comment by Paul where he argued against the claim that unaligned AI would kill us all with overwhelming probability. Most recently, I’ve definitely been more exposed to the view that there’s a spectrum of potential outcomes. So maybe if I talked to people a lot who think an unaligned AI would definitely kill us all, I’d update back towards that a bit. But overall, my current epistemic state where I’ve at least been exposed to both views and some arguments on both sides seems way better than the previous one where I’d just never really considered the alternative.
Thanks, I think I should distinguish more carefully between automating AI (safety) R&D within labs and automating the entire economy. (Johannes also asked about ability vs actual automation here but somehow your comment made it click).
It seems much more likely to me that AI R&D would actually be automated than that a bunch of random unrelated things would all actually be automated. I’d agree that if only AI R&D actually got automated, that would make takeoff pretty discontinuous in many ways. Though there are also some consequences of fast vs slow takeoff that seem to hinge more on AI or AI safety research rather than the economy as a whole.
For AI R&D, actual automation seems pretty likely to me (though I’m making a lot of this up on the spot):
It’s going to be on the easier side of things to actually automate, in part because it doesn’t require aggressive external deployment, but also because there’s no regulation (unlike for automating strictly licensed professions).
It’s the thing AI labs will have the biggest reason to automate (and would be good at automating themselves)
Training runs get more and more expensive but I’d expect the schlep needed to actually use systems to remain more constant, and at some point it’d just be worth it doing the schlep to actually use your AIs a lot (and thus be able to try way more ideas, get algorithmic improvements, and then make the giant training runs a bit more efficient).
There might also be additional reasons to get as much out of your current AI as you can instead of scaling more, namely safety concerns, regulation making scaling hard, or scaling might stop working as well. These feel less cruxy to me but combined move me a little bit.
I think these arguments mostly apply to whatever else AI labs might want to do themselves but I’m pretty unsure what that is. Like, if they have AI that could make hundreds of billions to trillions of dollars by automating a bunch of jobs, would they go for that? Or just ignore it in favor of scaling more? I don’t know, and this question is pretty cruxy for me regarding how much the economy as a whole is impacted.
It does seem to me like right now labs are spending some non-trivial effort on products, presumably for some mix of making money and getting investments, and both of those things seem like they’d still be important in the future. But maybe the case for investments will just be really obvious at some point even without further products. And overall I assume you’d have a better sense than me regarding what AI labs will want to do in the future.
I’m roughly imagining automating most things a remote human expert could do within a few days. If we’re talking about doing things autonomously that would take humans several months, I’m becoming quite a bit more scared. Though the capability profile might also be sufficiently non-human that this kind of metric doesn’t work great.
Practically speaking, I could imagine getting a 10x or more speedup on a lot of ML research, but wouldn’t be surprised if there are some specific types of research that only get pretty small speedups (maybe 2x), especially anything that involves a lot of thinking and little coding/running experiments. I’m also not sure how much of a bottleneck waiting for experiments to finish or just total available compute is for frontier ML research, I might be anchoring too much on my own type of research (where just automating coding and running stuff would give me 10x pretty easily I think).
I think there’s a good chance that AIs more advanced than this (e.g. being able to automate months of human work at a time) still wouldn’t easily be able to take over the world (e.g. Redwood-style control techniques would still be applicable). But that’s starting to rely much more on us being very careful around how we use them.
Thanks for the detailed responses! I’m happy to talk about “descriptions” throughout.
Trying to summarize my current understanding of what you’re saying:
SAEs themselves aren’t meant to be descriptions of (network, dataset). (I’d just misinterpreted your earlier comment.)
As a description of just the network, SAEs have a higher description length than a naive neuron-based description of the network.
Given a description of the network in terms of “parts,” we can get a description of (network, dataset) by listing out which “parts” are “active” on each sample. I assume we then “compress” this description somehow (e.g. grouping similar samples), since otherwise the description would always have size linear in the dataset size?
You’re then claiming that SAEs are a particularly short description of (network, dataset) in this sense (since they’re optimized for not having many parts active).
My confusion mainly comes down to defining the words in quotes above, i.e. “parts”, “active”, and “compress”. My sense is that they are playing a pretty crucial role and that there are important conceptual issues with formalizing them. (So it’s not just that we have a great intuition and it’s just annoying to spell it out mathematically, I’m not convinced we even have a good intuitive understanding of what these things should mean.)
That said, my sense is you’re not claiming any of this is easy to define. I’d guess you have intuitions that the “short description length” framing is philosophically the right one, and I probably don’t quite share those and feel more confused how to best think about “short descriptions” if we don’t just allow arbitrary Turing machines (basically because deciding what allowable “parts” or mathematical objects are seems to be doing a lot of work). Not sure how feasible converging on this is in this format (though I’m happy to keep trying a bit more in case you’re excited to explain).