MATS 9.0 with Patrick Butlin
Prev quant trader
@gilg_oscar
OscarGilg
Karma: 300
This seems fairly unlikely to me. This would be very selective and sophisticated sandbagging. I’m pretty confident that when the logs are full of Claude/Anthropic/AWS Astra mostly doesn’t think this is OpenAI. Worth running some follow-ups here though!
Yeah there are certainly a bunch of other capabilities involved in executing these audits, we don’t try to target all of them. My guess is model performance varies a lot across these different tasks. From using coding agents I’d guess models aren’t great at asking for more info.
We considered running the careful paraphrasing experiment! But on top of being costly it seems quite hard to get right. I suspect running smaller scale experiments is better to measure variability in self-incrimination rates bc of self-recognition.
How good are slop-vestigators?
Thanks! I haven’t listened yet but I think the iterative natural kind approach is quite similar indeed.
Agreed that you’re going to need to make judgement calls one way or another. But these don’t have to be final! You can also run follow-up investigations to gather new information. This should make judgement calls easier (but maybe bring up some new ones). One nice thing with models (compared to humans) is that it’s much easier to run experiments on them. E.g. we can resample and we have access to activations. So i would argue iterative approaches are less bottlenecked by judgement calls.
Thanks for the comment! I read 6.3. I definitely agree that we want to do Bayesian updating. I totally agree with:
And when you do this, you’ll need to think about how likely each theory of moral status is, and then how likely AIs are to have status conditional on your bottom up evidence and on that theory of moral status. In this way, there’s no substitute for grappling with questions about which theory of moral status is correct
However I disagree with this:
“and once you do that, you’re basically doing what you call ‘top down’ methodology”
It could be that saying “top-down” makes it sound stronger than we intend. We’re mainly making the case for far more “upwards” updating than past approaches. My view is roughly this:
My priors on theories of moral status/consciousness and their indicators are low. In particular it seems like Ai systems are too out-of-distribution for them (§1.2). Also i think there are complications with applying them (§1.1)
This changes how my Bayesian updates are likely to flow. The updates flow disproportionately “upwards” into revising theories compared to past approaches. Doing exploratory basic science is more useful under this scheme.
Maybe this is equivalent to what you mean by a top-down approach but with ~50% prior that all theories are wrong.
AI welfare research needs basic science
Ablating Split Personality Training
Yes, and the tasks are all prompts that actually ask the model to complete them. But indeed independent of any choice framing.
I was planning on doing my own fine-tuning to inject preferences, but this is a way easier way to de-risk!
It’s fairly in line with results in the post: the probes consistently transfer better than the utility correlation would predict, but this works less well on personas that are more different from the assistant.
I’m still pending approval for the misalignment one...
Thanks for the comment!
So this gets to a slightly unintuitive part of my setup that I maybe didn’t explain well enough. The probe scores are a function of a single task’s activations. They are trained to predict a given task’s score based on activations from that single task prompt.
So there is in fact no mechanism for the probe to do relative scoring between two tasks in a given pairwise choice. I just use pairwise choices because I think this is (by far) the best way to measure preferences in LLMs. I’ll add a footnote or maybe emphasise it more, this feature of the setup is non-trivial.
You’re right.
I tried exactly that, result below. Using the Qwen3 embeddings gets you almost all the way to the pre-trained model performance. The per-topic breakdown is also pretty similar to the PT model.
Regarding using activations from a more capable model, since models have similar preferences it would be quite hard to draw strong conclusions on the methodology. But I’m going to try something like the AA > BA idea in your other comment, using sufficiently different models.
I defined it as
roughly internal states that encode “how much do i want this?” and play some role in driving choice
(note that I just added the bold because i agree the definition wasn’t standing out).
To put it more concretely you can decompose it into two separate conditions 1) it encodes some quantifiable value for the system 2) it plays a causal role in how the system selects actions.
It’s not the most commonly used term in philosophy… But the choice is deliberate! E.g. using a word like “valence” would be tying in a bunch of other assumptions
I used the model <start_of_turn> token (in fact I will edit the post to make that clearer).
So for base models I was actually just using the actual last token in the prompt (not a turn boundary token). I definitely should make this clearer.
I’m going to try using the mean across the whole prompt. Although even if I found that instruct probes outperformed pre-trained model probes, it doesn’t rule out a more mundane explanation where the post-training is just teaching the model to aggregate information at the turn-boundary tokens. Do you agree? Thanks for the comments btw!
Models have linear representations of what tasks they like
Exactly! And then also a decent amount of probability mass on none of our candidate explanations being remotely close to the truth :)
It’s maybe worth considering that our intuitions about the “On the one hand” argument might be flawed. You could make the case that this “unaltered latent state” ideal is misguided, because the attention mechanism works well precisely using both the K and Q degree of freedom.
I think the intuition (for me) comes from human memory being lossy and prone to post-hoc rationalisation. Models clearly sometimes do this too. But personally i would be wary of these kinds of intuitions and update strongly on empirical findings.
It would also be interesting to see if the steering vectors you extract looks like a persona vector https://www.anthropic.com/research/assistant-axis. Or maybe baseline our method with steering using out of the box “honest” persona vectors.
I had similar intuitions. Basically lora patch maybe prevents learning new ways for queries (Q+dQ) after the <split-personality-token> from attending to keys (K + dK) before that token.
So the hypothesis is that the dK^T * Q term is doing important work. Here is an attempt to flesh it out in empirical predictions.
Important to note that we are currently doing LoRA to MLP layers too. Not a big issue for the argument imo.
We could try training LoRA on everything except K. Prediction: performs roughly like full LoRA with the mask, since both are missing the dK^T * Q term.
Then we could try K-only LoRA (or K + MLP). Then masking should severely harm performance. Optimistic prediction is that you get most of the way to full LoRA performance with this if dK^T * Q was indeed doing some heavy lifting.
Having said that I don’t have super strong intuitions for dismissing the dK^T*dQ term out of hand. If that term were doing the job it would seem less interesting than if dK^T was single-handedly amplifying the signal.
Models would probably do quite well with narrow prompts like True/False checks. We were considering incorporating this, but ultimately prioritised the arguably more realistic open-ended setting, with no ex ante knowledge of the incident. We think identifying which findings are actually worth reporting is very important here.
It might also be very useful to have investigator agents which already have some big picture understanding of the incident and go collect evidence. My guess is agents are already pretty good at this and it’s not the bottleneck.