Research Scientist working on Alignment Red Teaming team at the UK AI Security Institute
RobertKirk
Karma: 500
I’m not convinced by the argument that a lot of these black-box evals would be ecologically valid (without significant effort to make them so). I think models could plausibly see through you saying it will be monitored closely/reviewed by humans, and potentially end up not acting as if that is true; or, including that statement may trigger it to believe its alignment may be being tested.
For the former, see the example here: https://www.lesswrong.com/posts/LhXW8ziwnn7Dd8edm/a-toy-environment-for-exploring-reasoning-about-reward#The_model_is_increasingly_insensitive_to_threats_of_auditing_or_human_review – in a toy reward-hacking environment a capabilities-only version of o3, even when told explicitly it’s responses will be reviewed if it reward-hacks, still ends up reward-hacking, and “it often seemed to reason itself into the idea that the threat of review was just a bluff”.
I do think you’d be able to track if these things are happening somewhat by reading the reasoning, but I think that the naïve method for getting models to believe something, so you can see how they behave under that belief, isn’t likely to work reliably enough to be sure any difference in behaviour is due to a difference in the models’ beliefs. This is the same argument laid out in https://www.lesswrong.com/posts/3HeauQLSHosRiwyto/measuring-reward-seeking-via-contrastive-belief-updates-1 for why they use SDF rather than just prompting for belief modification.
I did some related work with mentees last year that extends EDW by using an unsupervised optimisation target: the average CE loss of an LLM on items in the dataset when given the featurisation of the item in the prompt (https://arxiv.org/abs/2502.17541). That kind of approach can help you prioritise the features which explains the dataset you’re investigating the most (kind of a natural language PCA), and can be applied to unstructured data. I think stuff like this is pretty interesting and would be excited to see more of it!
I like your suggestion of a proxy task also, although likely the it can be optimised directly (i.e. selecting over documents that have high score on that task in various ways); I don’t know if that diminishes it’s usefulness.
Several frontier models are substantially prefill aware
Tracing Eval-Awareness Emergence Through Training of OLMo 3
This is pretty similar to ideas you can use for eval awareness (i.e. prompt the model to output some token if it thinks it’s in an eval). in that setting, i’ve generally seen similar results there compared to direction asking the model. I should note also we’re mostly not using reasoning here (I believe), so the model doesn’t have many tokens to reason with before outputing it’s prediction.
It’s also not clear which (if any) of these methods are upper or lower bounds and whether they are tight or not. e.g. your idea could give the model more opportunity to reason about it’s answer—it knows from the beginning of it’s output that it should be thinking about whether it’s prefilled, so it can think about it in it’s CoT and then decide whether to output <INJECTED> or not.
In general because of this uncertainty, I’m in favouring of throwing all reasonable measures at this and seeing what they say, trying to minimise all of them when designing prefills/evals, and trying to reason about what any remaining awareness might mean for your evaluation.
If you find this kind of work interesting, the alignment red team (which Alex and I are part of) are hiring currently: https://job-boards.eu.greenhouse.io/aisi/jobs/4784438101
Prefill awareness: can LLMs tell when “their” message history has been tampered with?
A Sober Look at Steering Vectors for LLMs
I think this and other potential hypotheses can potentially be tested empirically today rather than only being distinguishable close to AGI
How would you imagine doing this? I understand your hypothesis to be “If a model generalises as if it’s a mesa-optimiser, then it’s better-described as having simplicity bias”. Are you imagining training systems that are mesa-optimisers (perhaps explicitly using some kind of model-based RL/inference-time planning and search/MCTS), and then trying to see if they tend to learn simple cross-episode inner goals which would be implied by a stronger implicity bias?
A quick technical question: In the comparison to fine-tuning results in Section 6 where you stack CAA with fine-tuning, do you find a new steering vector after each fine-tune, or are you using the same steering vector for all fine-tuned models? My guess is you’re doing the former as it’s likely to be more performant, but I’d be interested to see what happens if you try to do the latter.
I think a point of no return exists if you only use small LRs. I think if you can use any LR (or any LR schedule) then you can definitely jump out of the loss basin. You could imagine just choosing a really large LR to basically resent to a random init and then starting again.
I do think that if you want to utilise the pretrained model effectively, you likely want to stay in the same loss basin during fine-tuning.
To check I understand this, is another way of saying this that in the scaling experiments, there’s effectively a confounding variable which is model performance on the task:
Improving zero-shot performance decreases deletion-CoT-faithfulness
The model will get the answer write with no CoT, so adding a prefix of correct reasoning is unlikely to change the output, hence a decrease in faithfulness
Model scale is correlated with model performance.
So the scaling experiments show model scale is correlated with less faithfulness, but probably via the correlation with model performance.
If you had a way of measuring the faithfulness conditioned on a given performance for a given model scale then you could measure how scaling up changes faithfulness. Maybe for a given model size you can plot performance vs faithfulness (with each dataset being a point on this plot), measure the correlation for that plot and then use that as a performance-conditioned faithfulness metric? Or there’s likely some causally correct way of measuring the correlation between model scale and faithfulness while removing the confounder of model performance.
If you train on infinite data, I assume you’d not see a delay between training and testing, but you’d expect a non-monotonic accuracy curve that looks kind of like the test accuracy curve in the finite-data regime? So I assume infinite data is also cheating?
I’ve been using “delayed generalisation”, which I think is more precise than “grokking”, places the emphasis on the delay rather the speed of the transition, and is a short phrase.
Speculative inferences about path dependence in LLM supervised fine-tuning from results on linear mode connectivity and model souping
I think the hyperlink for “conv nets without residual streams” is wrong? It’s https://www.westernunion.com/web/global-service/track-transfer for me
This feels kind of like a semantic disagreement to me. To ground it, it’s probably worth considering whether further research on the CCS-style work I posted would also be useful for self-driving cards (or other applications). I think that would depend on whether the work improves the robustness of the contrastive probing regardless of what is being probed for (which would be generically useful), or whether it improves the probing specifically for truthfulness in systems that have a conception of truthfulness , possibly by improving the constraints or adding additional constraints (less useful for other systems). I think both would be good, but I’m uncertain which would be more useful to pursue if one is motivated by reducing x-risk from misalignment.
I think that “don’t kill humans” can’t chain into itself because there’s not a real reason for its action-bids to systematically lead to future scenarios where it again influences logits and gets further reinforced, whereas “drink juice” does have this property.
I’m trying to understand why the juice shard has this propety. Which of these (if any) are the the explanation for this:
Bigger juice shards will bid on actions which will lead to juice multiple times over time, as it pushes the agent towards juice from quite far away (both temporally and spatially), and hence will be strongly reinforcement when the reward comes, even though it’s only a single reinforcement event (actually getting the juice).
Juice will be acquired more with stronger juice shards, leading to a kind of virtuous cycle, assuming that getting juice is always positive reward (or positive advantage/reinforcement, to avoid zero-point issues)
The first seems at least plausibly to also to apply to “avoid moldy food”, if it requires multiple steps of planning to avoid moldy food (throwing out moldy food, buying fresh ingredients and then cooking them, etc.)
The second does seem to be more specific to juice than mold, but it seems to me that’s because getting juice is rare, and is something we can better and better at, whereas avoiding moldy food is something that’s fairly easy to learn, and past that there’s not much reinforcement to happen. If that’s the case, then I kind of see that as being covered by the rare-states explanation in my previous comment, or maybe an extension of that to “rare states and skills in which improvement leads to more reward”.
Having just read tailcalled comment, I think that is in some sense another of phasing what I was trying to say, where rare (but not too rare) states are likely to mean that policy-caused variance is high on those decisions. Probably policy-caused variance is more fundamental/closer as an explanation to what’s actually happening in the learning process, but maybe states of certain rarity which are high-reward/reinforcement is one possibly environmental feature that produces policy-caused variance.
So I struggle to think of an example of a tool that is good for finding insidious misaligning bugs but not others.
One example: A tool that is designed to detect whether a model is being truthful (correctly representing what it knows about the world to the user), perhaps based on specific properties of truthfulness (for example see https://www.alignmentforum.org/posts/L4anhrxjv8j2yRKKp/how-discovering-latent-knowledge-in-language-models-without) doesn’t seem like it would be useful for improving the reliability of self-driving cars, as likely self-driving cars aren’t misaligned in the sense that they could drive perfectly safely but choose not to, but rather are just unable to drive perfectly safely because some of their internal (learned) systems aren’t sufficiently robust.
It would be interesting to compare various black-box methods for getting models to believe various things and see which work best (as measured either by held-out black-box belief elicitation or white-box methods). I can see the AGENTS.md, subagents or COT editing all being better than user prompt editing, but would be curious to see which works best (of course, this requires reliable white-box belief estimation methods, but I think you could still do this on a range of simpler tasks where we’re more confident they work, and then extrapolate).