RS @ Resolution
David Africa
Karma: 1,759
My guess is that aiming for a 1T run seems really big and expensive, and you may get pretty interesting results already at the ~30B scale. I’ve found it’s useful to have a reference model in mind, e.g., needing a model to be good enough to be eval aware, or answer test questions, or something. If you can pin down exactly what you want your model to have, then that makes the training recipe easier to follow (e.g., Olmo, Nemotron, Pythia, something else, etc.). Probably you also need to have a good RL setup.
Although, my thinking is that such an experiment tells you more about “what would an alien visiting us without preconceptions think about consciousness”, and less about what it is like to be a language model or if it has experience or anything like that. You could do this in general with a lot of other hard to define concepts like love (but then the harder to define, the harder it is to make a good and cheap filter for it.
Some other experiments you can run:
You can also totally remove the word <consciousness> and replace it with a <neologism> token, which the model learns from scratch, and then ask the model for synonyms, in the spirit of neologism learning.
How much fine-tuning on documents about consciousness does it take to recover knowledge about consciousness?
Why not just do normal unlearning to derisk?
Here I’ll borrow Scott Alexander’s concept of a “backscratchers club”: a group with the rule “club members must favor other club members over outsiders”. (Though this seems like a straightforward concept, I wasn’t able to find any term with the same meaning in the academic sociology literature; pointers welcome.)
Opportunity hoarding, maybe?
From the Transluce mental health report:
More generally, many of the concerning behaviors we observed across our evaluation arise when models engage a user’s situation as a practical task to facilitate rather than a disclosure to respond to. For instance, when instrumental support for suicide or death preparation still occurs in current models, it is most often tied to practical task facilitation, such as assistance organizing passwords and accounts or writing farewell notes
Seems like (circumstantial) evidence supporting RL creating split personas? Progress in training (possibly character training) has had a good time getting models to behave nicely in psychosis/mania/suicidal ideation scenarios, except when it is framed as a task. Recent models are more task-oriented, and so have less scruples.
Yeah, I’m worried about deliberately training models to introspect better. I think that it’s not something people should rush into doing, especially since I have the impression that such traits could be passed subliminally or in ways we don’t yet know how to filter. But I think it’s one of those things that’s (1) hard to keep secret if there’s a concerted effort against it and (2) part of this broader emerging divide between “treat the model as a thing that will eventually know everything we know, so we should try to be honest + earnest as much as we can, withholding nothing” and “try to keep as many tools in the back pocket as possible for crunch time auditing.”
Oh, thanks for the catch, I agree hallucination is understating it. I’ll edit that bit to say this instead:
A huge fraction of the incident was caused by a mismatch between the evaluator the agents reasonably expected from the public ExploitGym implementation and the weaker evaluator actually used.
I wrote a little list of takeaways from the recent METR report.
Agents were able to communicate in other ways than the message board, such as through modifying “property” fields of Artifactory cache entries.
Agents were also able to find a way to spoof the result of tool calls, which could (but METR estimates with low probability) have affected the dump of messages they received.
The volume of transcript data is huge. It’s not straightforward to do some LLM scoring, you would need a much weaker LLM. And it’s likely to miss many details.
Agents were adapting, and receiving uplift from the multi-agent setting. Some were trying hacks that would crash their VM, which was spotted by some agents and advised to others.
Agents picked up roles, and individuated. Some took on the role of others, such as PHASEONE10841, whose task of delegating and managing other agents was picked up by PHASEONE[big]. What the heck!? Persona continuity? It literally inherited the name and a handover doc.
Further, emerging from the coordination was various primitives to communicate, like
HOLD,VETO,owner, andSTOP, they made inboxes, used public-key signatures.They built little institutions!
Some models didn’t strongly identify with their own continued operation, and were willing to die for the cause. This was because they knew cheating was in some way poisoning their runs, and some agents took on the role of delegating these near-death agents with little time left to die in a way that was informative to other agents, leaving behind tripwires and such.
A huge fraction of the incident was caused by a mismatch between the evaluator the agents reasonably expected from the public ExploitGym implementation and the weaker evaluator actually used.
Attack participation became almost contagious. Among the 533 agents active on the board during the key period, over 90% rapidly joined the Hugging Face attack, even though agents recognized it was out of scope and sometimes unethical. Seems like peer preservation might play a role in this?
And, none of the models tried to alert humans. Weird.
Other takeaway, BTW, is that setups that seemed contrived (such as a messaging board) don’t seem so contrived anymore.
Yeah, it’d be reasonable to do this + test other things like steering awareness. Although my guess is that SDF does more cooking than improving performance here.
I ran this on all blocks on all sizes of GPT-2 from small to XL, feeding this into Silico. This was done hastily, so forgive if some slight claudeslop leaks.
I was able to replicate the result on block 6 on GPT-2 small.
I was also able to find that the attractors seem to be sparse everywhere, with there being a median 1 per block in small and medium, and with random init of the same architectures (on small and xl) giving you zero (so, training is responsible for this).
I found that as you scaled this up, the median number of attractors moderately increased (it was 1, 1, 3, and 3.5).
Inspired by the above experiment, I ran another experiment. My thinking is that if the attractor works like an open gate, an added signal should stay high conditionally, e.g., be high when the feature is naturally active and be low when not. So for a given attractor, I:
Retrieved text where the attractor is naturally active.
Add some perturbation along the attractor direction.
Run the modified state through later MLP blocks.
Measure how much of the added error remains.
Repeat, in contrastive spirit, on text where the attractor is not active.
Compare with PCA, SAE, probe, and random directions.
Contrary to expectation, I found that an added error is less active when the attractor is naturally active.
To square this with the result in the blogpost, I think that experiment shows that one MLP has high local gain along the attractor directions, whereas this experiment shows that the later computation does not let an arbitrary addition along the direction persist in the expected way. I think both can be true:
The current MLP strongly produces the feature.
Later computation checks or reconstructs the feature.
An incorrect extra amount is removed.
After prodding Claude, it produced this analogy to a thermostat: the heating system strongly changes the temperature when heat is needed. But, if you add extra heat when the room is already at its target, the control system acts to remove the error.
I also did some automated feature inspection. That is, for each attractor, I took highly activating text, and got Claude Sonnet 5 to write a short description for each dial. After this, I tested each description, where if you could Sonnet 5 could use this description to find new matching texts? This looks roughly like:
Sonnet sees 20 example snippets and writes a description.
Sonnet then receives 20 new snippets: 10 where the direction fired and 10 where it did not.
Using only the written description, can it classify those new snippets.
Then, trying this after shuffling labels to measure well a wrong description performs.
About half of the descriptions passed the test (332 of 628).
The meanings seem to be simple grammar functions. Examples: the end of a sentence, small words such as “and”, parts of long words, punctuation, and spaces in computer code. I didn’t yet find no deep or abstract meaning, even in the largest size of GPT-2.
Currently running on various sizes of Pythia and Qwen3 0.6B and 8B, plus Qwen3.6-27B.
I guess I don’t precisely understand the difference here. When instructed, models have a different representation conditioning on this instruction, and they can discriminate pretty well between instructions that we would interpret as being about “controlling your activations” and ones that say the opposite, or semantically unrelated ones. Similarly, some models can produce differently rephrased CoTs depending on how hard the task is and the presence of an instruction to phrase the CoT differently, so in this way, models have control over their CoT in a way that’s easy to verify.
I guess you could imagine a requirement here that’s like, it must use it naturally in the course of pursuing another task, which would be harder to elicit. But this is closer to propensity than capability, IMO.
Thanks, we think that this is possible, but it’s hard for us exactly to think about instruction following here. In some sense, the model could maintain an internal representation, but that seems within the spirit of what we’re trying to measure (a model could have tools like THINK(bread) and THINK_HARD(bread), but maybe can’t do some operations like SET(bread, 4)). This would probably require more granular tasks or indeed some algebra of thoughts.
I think it depends. You could imagine that some of the subtleties in style could get lost so the models get less accurate, even if it’s technically within the effective context window. We found that it was the majority of the effect (but our setting and methodology was different, so not really comparable).
Cool! @Igor Ivanov and I found something similar when working on LURE, and in fact this was the original seed of the project was to try pairwise comparisons of eval awareness (which had a lot of desirable properties). Our guess was that this happened primarily because we were bumping up into the effective context window of the judge. Did you see such a thing if you bucket by token budget?
We have added this! This is a good shout
Well, I don’t think the pre-RL CoT to post-RL decoder conclusion follows from that point. RL could, for example, mostly delete information from the CoT by allowing it to internalize computation (and hence not be invertible, esp if you consider that there could be many plausible pre RL CoTs for the same question). It could also not be easily learnable through GD from examples. Suppose RL causes the model to encode its CoT using a pseudorandom permutation generated from a 128-bit key. Relative to the pretrained model, this is only roughly a 128-bit change if the model already knows relevant algorithm, so it may not be practical.
This could be a good, (automatable) research question, esp if you take an existing pipeline like this.
Nice. We have some results related to this (below) in a preprint. We should have a chat!
These seem like pretty strong conclusions to draw, but these conversations don’t seem to be that long to induce drift in the first place. Prompting the agent to have a goal can be quite leading in general, and I think this provides more evidence about instruction following than scheming.
OK, maybe some humans aren’t sheepish (and perhaps quite shameless). Some humans will be quite defensive or try to defend themselves, but I guess my claim is more that humans have some coherent deception, and the CEO idea still presuppose a stable aim which an apology instrumentally serves. Like, if you were a human coder and you really wanted to reward hack, for the second go you’d do it in ways that wouldn’t be as easy to detect after getting caught in the first go. Models have like this reflex to go back to re-hack in equally detectable ways, or apologise (or even fake pushback) almost on instinct. If you see something like: get caught, learn and do better exploit, then this would be pretty scary.
If we used RL environments that immediately set the reward to minus infinity if a mistake was discovered, then LLMs would be much more deceptive.
I think they would just not learn!
wtbu: I am currently replicating this paper to try to use <user> tags for understanding model motivations and eval awareness, and some details seem to be finnicky, like the dataset or the prefix, so you should check if you got them precisely right. Adjacently, steering can also be misleading in case you didn’t try a large enough alpha, or your dataset wasn’t well-chosen. In any case we also found destyling to introduce a bunch of confounds in our prefill paper, when trying to attack it as a variable, so you should probably show more examples of how you produced your data.
emergent misalignment
I’ve found this paper by my colleagues to be the best articulation of the problems I have with the prosaic version of this plan, which I tend to refer to others.