MATS 9.0 with Patrick Butlin
Prev quant trader
@gilg_oscar
OscarGilg
Karma: 300
This seems fairly unlikely to me. This would be very selective and sophisticated sandbagging. I’m pretty confident that when the logs are full of Claude/Anthropic/AWS Astra mostly doesn’t think this is OpenAI. Worth running some follow-ups here though!
Yeah there are certainly a bunch of other capabilities involved in executing these audits, we don’t try to target all of them. My guess is model performance varies a lot across these different tasks. From using coding agents I’d guess models aren’t great at asking for more info.
We considered running the careful paraphrasing experiment! But on top of being costly it seems quite hard to get right. I suspect running smaller scale experiments is better to measure variability in self-incrimination rates bc of self-recognition.
Thanks! I haven’t listened yet but I think the iterative natural kind approach is quite similar indeed.
Agreed that you’re going to need to make judgement calls one way or another. But these don’t have to be final! You can also run follow-up investigations to gather new information. This should make judgement calls easier (but maybe bring up some new ones). One nice thing with models (compared to humans) is that it’s much easier to run experiments on them. E.g. we can resample and we have access to activations. So i would argue iterative approaches are less bottlenecked by judgement calls.
Thanks for the comment! I read 6.3. I definitely agree that we want to do Bayesian updating. I totally agree with:
And when you do this, you’ll need to think about how likely each theory of moral status is, and then how likely AIs are to have status conditional on your bottom up evidence and on that theory of moral status. In this way, there’s no substitute for grappling with questions about which theory of moral status is correct
However I disagree with this:
“and once you do that, you’re basically doing what you call ‘top down’ methodology”
It could be that saying “top-down” makes it sound stronger than we intend. We’re mainly making the case for far more “upwards” updating than past approaches. My view is roughly this:
My priors on theories of moral status/consciousness and their indicators are low. In particular it seems like Ai systems are too out-of-distribution for them (§1.2). Also i think there are complications with applying them (§1.1)
This changes how my Bayesian updates are likely to flow. The updates flow disproportionately “upwards” into revising theories compared to past approaches. Doing exploratory basic science is more useful under this scheme.
Maybe this is equivalent to what you mean by a top-down approach but with ~50% prior that all theories are wrong.
Yes, and the tasks are all prompts that actually ask the model to complete them. But indeed independent of any choice framing.
I was planning on doing my own fine-tuning to inject preferences, but this is a way easier way to de-risk!
It’s fairly in line with results in the post: the probes consistently transfer better than the utility correlation would predict, but this works less well on personas that are more different from the assistant.
I’m still pending approval for the misalignment one...
Thanks for the comment!
So this gets to a slightly unintuitive part of my setup that I maybe didn’t explain well enough. The probe scores are a function of a single task’s activations. They are trained to predict a given task’s score based on activations from that single task prompt.
So there is in fact no mechanism for the probe to do relative scoring between two tasks in a given pairwise choice. I just use pairwise choices because I think this is (by far) the best way to measure preferences in LLMs. I’ll add a footnote or maybe emphasise it more, this feature of the setup is non-trivial.
You’re right.
I tried exactly that, result below. Using the Qwen3 embeddings gets you almost all the way to the pre-trained model performance. The per-topic breakdown is also pretty similar to the PT model.
Regarding using activations from a more capable model, since models have similar preferences it would be quite hard to draw strong conclusions on the methodology. But I’m going to try something like the AA > BA idea in your other comment, using sufficiently different models.
I defined it as
roughly internal states that encode “how much do i want this?” and play some role in driving choice
(note that I just added the bold because i agree the definition wasn’t standing out).
To put it more concretely you can decompose it into two separate conditions 1) it encodes some quantifiable value for the system 2) it plays a causal role in how the system selects actions.
It’s not the most commonly used term in philosophy… But the choice is deliberate! E.g. using a word like “valence” would be tying in a bunch of other assumptions
I used the model <start_of_turn> token (in fact I will edit the post to make that clearer).
So for base models I was actually just using the actual last token in the prompt (not a turn boundary token). I definitely should make this clearer.
I’m going to try using the mean across the whole prompt. Although even if I found that instruct probes outperformed pre-trained model probes, it doesn’t rule out a more mundane explanation where the post-training is just teaching the model to aggregate information at the turn-boundary tokens. Do you agree? Thanks for the comments btw!
Exactly! And then also a decent amount of probability mass on none of our candidate explanations being remotely close to the truth :)
It’s maybe worth considering that our intuitions about the “On the one hand” argument might be flawed. You could make the case that this “unaltered latent state” ideal is misguided, because the attention mechanism works well precisely using both the K and Q degree of freedom.
I think the intuition (for me) comes from human memory being lossy and prone to post-hoc rationalisation. Models clearly sometimes do this too. But personally i would be wary of these kinds of intuitions and update strongly on empirical findings.
It would also be interesting to see if the steering vectors you extract looks like a persona vector https://www.anthropic.com/research/assistant-axis. Or maybe baseline our method with steering using out of the box “honest” persona vectors.
I had similar intuitions. Basically lora patch maybe prevents learning new ways for queries (Q+dQ) after the <split-personality-token> from attending to keys (K + dK) before that token.
So the hypothesis is that the dK^T * Q term is doing important work. Here is an attempt to flesh it out in empirical predictions.
Important to note that we are currently doing LoRA to MLP layers too. Not a big issue for the argument imo.
We could try training LoRA on everything except K. Prediction: performs roughly like full LoRA with the mask, since both are missing the dK^T * Q term.
Then we could try K-only LoRA (or K + MLP). Then masking should severely harm performance. Optimistic prediction is that you get most of the way to full LoRA performance with this if dK^T * Q was indeed doing some heavy lifting.
Having said that I don’t have super strong intuitions for dismissing the dK^T*dQ term out of hand. If that term were doing the job it would seem less interesting than if dK^T was single-handedly amplifying the signal.
Thanks for sharing, I hadn’t seen it personally.
Yes, the main difference is that we train separate LoRA weights through SFT, whereas they do RL on the same model (but with a separate reward model). In the limit I would guess something like what they did might work better. But some nice things about the LoRA + SFT method are that it is very cheap and keeps the main model exactly the same.
Thats what illusionist mean by illusionism, but you haven’t offered much motivation to believe it, and it needs motivation, because its far from obvious:-
Sure, right now I haven’t offered that much motivation. The post is already probably too long.
The motivation, therefore is a strong belief in physicalism. Note that the basic manouvre—saying that if E is apparently evidence against hypothesis H , E cannot be true—can be generalised to other areas , and can be used to “prove” almost anything
Agreed that this would be very bad reasoning. But some inference can still be drawn right. If you have very strong independent priors for H and I show E is apparently evidence against H then your credence in E might go down (as might your credence in H to be fair). That would be one prima facie motivation.
Illusionism , as opposed to delusionism, has the further problem Th t it explains the illusion of Qualia as a quasi-phenomenal property: so it eliminates the kind of phenomenal properties for which there is direct evidence in favour of another kind for which there is none.
This is where the big disagreement happens imo. Because illusionists would say the data to explain is “reports about qualia” and not “qualia” (Dennett spoke about heterophenomenology). And for that we have a lot of evidence.
Circling back to motivations, one I didn’t mention in the post is unreliability of introspection undercutting reasons to believe our intuitions about consciousness. A great book, Eric Schwitzgebel’s Perplexities of Consciousness, talks about this e.g. “do we dream in colours?” “can humans echolocate?” “Do you constantly experience your feet in your shoes?”.
One final prima facie motivation is that we should not necessarily expect evolution to produce transparent introspective access to our cognitive processes. Obviously there is some information being exchanged, but the representation only needs to be as faithful as survival demands.
Greatly appreciate the comment. I agree with most of it. As for the white light analogy, I’m definitely updating towards being less confident about it. Here is a perhaps stronger way of reframing it that I would be keen to get thoughts on:
Because the explanation of white light doesn’t tell you that there is no such thing, it just tells you that it is made of parts, non fundamental. (Confusion between reduction and elimination is rife).
So if I understand correctly, you’re saying that illusionism wants to eliminate phenomenal consciousness but that here I’m using a reduction analogy with white light. But I think both white light and consciousness deserve both treatments depending on what exactly we’re targeting.
Of course white light is real, and can be reduced to xyz. But now consider something like (and this is where I’m somewhat moving the goalposts compared to what I originally wrote) “whiteness as pure luminance”, then that can be eliminated. The analogy was chosen because pre-Newton white light was considered the purest, most fundamental form of light—with colours thought to be modifications or corruptions of this pure white light
For consciousness, illusionists seek to eliminate phenomenal consciousness but simply reduce so-called intuitions about consciousness. Eliminate the hard problem but simply reduce the meta-problem.
Thoughts?
Of course phenomenal consciousness is a model of the outside world: testable rd presents certain frequencies of light, and so on.
I’m not sure. The way I roughly see it, when you see red you have A) a representation that tracks red wavelengths and enables red-appropriate behaviours B) a representation of some additional “reddish feel”. The first one definitely models the outside world but the second is a systematic misrepresentation.
Lots of great ideas. I like the approach of systematically listing out properties that consciousness might have, especially for studying LLMs.
But here’s my issue: most of the properties you listed don’t go any way towards explaining “why we think we are conscious”. You could argue perhaps self-perception and theory of mind do that, but form how you later fill in the boxes for e.g. “countries”, I don’t think those are the properties we need. It would be quite weird if our theory of consciousness didn’t explain why we even came up with the concept. Or at the very least what consciousness does, its causal impact. Michael Graziano calls this the Arrow B problem.
You could refine your set of properties to tackle that, but then I’m not sure the boxes for “country” or “Wildfire in LA” can be filled!
Obviously you could say maybe consciousness doesn’t do anything, has no causal impact: it’s an epiphenomenon. But it clearly has at least one impact: it’s making us talk about it right now. Or maybe it’s not an epiphenomenon in some entities (e.g. humans), but it is an epiphenomenon in other entities. I’ll have to think about that but it feels like you would need a very principled reason to believe this combination of things.
Models would probably do quite well with narrow prompts like True/False checks. We were considering incorporating this, but ultimately prioritised the arguably more realistic open-ended setting, with no ex ante knowledge of the incident. We think identifying which findings are actually worth reporting is very important here.
It might also be very useful to have investigator agents which already have some big picture understanding of the incident and go collect evidence. My guess is agents are already pretty good at this and it’s not the bottleneck.