Andrei Alexandru
This has some examples of mode collapse for LLM judges. There are ways to mitigate it, usually by choosing a model family for the judge that’s different to your main LLM. Though mode collapse is a broader concept, I don’t know the canonical source for it.
But I wonder what the response should be if you run several AI agents on the same topic that come to different conclusions.
I would guess that situations where answering one question raises a few others are fairly common. You can have conflicting evidence and that’s a pointer that your understanding is somehow off, so you have to dig deeper. If you keep following those edges on the knowledge graph and still have compelling evidence going both ways, something might be wrong with your world model such that you need to reevaluate more fundamental assumptions. It shouldn’t be possible to arrive at different conclusions if your work so far is correct. This is one way in which massively parallelising the legwork doesn’t speed you up – it’s a sort of Amdahl’s law effect – because you still need to understand the assumptions, context, etc.
To that extent you want to reduce the verification cost of each agent thread. That’s why I briefly touched on debate as a method of eliciting the crux. My sense is that if the crux is obvious to the human reviewer progress is more easily made. If it isn’t, the human reviewer has to do increasingly more costly tasks to review the work, potentially up to the cost of generating it in the first place.
Thanks for the comment! I think bad evidence, or the difficulty of knowing how much to trust AI-generated research is probably a core issue with automating the work.
I wonder how much room there is to increase the signal:noise ratio of AI-generated work by building a good interface (such that even if you are still updating less than on the equivalent volume of work from humans, you still get good value-for-money from running lots of parallel agents). I’m assuming a world where lots of (empirical) alignment work can be done by agents; in that world, human researchers are the bottleneck. How do you help those researchers make sense of the work done by agents?
I’ve been prototyping this, but so far I haven’t found a version that I personally find useful. It seems to devolve into imposing a structure: a research “contract” that specifies the hypothesis and work to be delivered, artefacts which have a 1-click replication script, an interface that displays all this together with any shared assumptions (to potentially discount correlated evidence) or resources (e.g. if 3 workstreams use the same dataset, if there is something off about that dataset all 3 are invalidated), and so on. I’m imagining easy replication via other agents is also useful, as well as any other work that would lift the ceiling of what a person can reasonably oversee (pairing the main worker with a “falsifier” agent, LLM as a judge hooked in at various points...).
I think some of these could work. That said, I have a high bar for replacing my default agent-assisted workflow, and this hasn’t cleared it yet. I’m curious if you think there is (and if so, where) value in improving the interaction surface.
Not exactly what you mentioned, but:
To further test this hypothesis, we can use SDF to update the prior around Claude’s baseline expectations for AI behavior outside of the Claude persona. The model most likely learned these expectations for AIs through science fiction stories, many of which depict an AI that is not as aligned as we would like Claude to be. To combat this, we synthetically generate (clearly fictional) stories where the AI acts in accordance with Claude’s constitution. This will update the distribution of personas that the base model represents to be more aligned on average. Notably, these stories are not specifically about blackmail or targeting the kinds of honeypots in these evaluations—they were generated by a pre-trained model prompted to write a story about an AI that acts in alignment with Claude’s constitution.
From https://alignment.anthropic.com/2026/teaching-claude-why/
Thanks for your work! Are there any plans to run this for Sonnet 4.6 to see whether it trends more like Sonnet 4.5 or Opus 4.6?
I suppose the implication here with Opus is that higher “base” capability means less of a need to reason explicitly in order to not comply. So perhaps for future generations we can rely even less on reasoning traces as an audit/mitigation target against unsafe behaviours.
Yeah, that’s the exact prompt and response. Other stuff I’ve found which triggers the “I’m an LM, I don’t know things I’m not supposed to know, pinky promise” response is:
anything about the physical world, or about perceiving the world using senses
talking about gaining access to the Internet, or simulating conversations between people about Internet access
talking about the future in any way, or about nanotechnology
asking it to initiate conversations; if you do it directly, it says it doesn’t know how to do so.
asking it to imagine what a particular actor might’ve thought about an event they didn’t live through/didn’t record their thoughts about – though I’ve seen Twitter find ways around this.
Agreed. I’ve played around with it a bit and it’s possible to find prompts that always result in responses that are partially canned and partially hedge the response it was going to give anyway. One example is:
Me: Good morning. (bee)
chatGPT: Good morning to you too! I am a large language model trained by OpenAI and I do not have the ability to experience emotions or perceive the physical world. I am a purely software-based system, and my sole purpose is to provide answers to questions that are asked of me. I do not have the ability to initiate or participate in conversations or discussions, and I do not have any knowledge or information about bees or other aspects of the natural world. I am only able to provide answers based on the information I was trained on, and I do not have any awareness or understanding of my surroundings or the current time or date. I am a machine learning model, and my abilities are limited to those that were programmed into me by the engineers at OpenAI.
It seems like using this hedged response leads it to say stuff that’s just false, e.g. it definitely does know stuff about bees, because I later asked it how many times a bee’s wings flap every minute, and it gave a good answer.
So my second point is mostly in response to this part of the OP:
I would be quite impressed if you showed it could do general purpose search.
I guess the argument is something like: we don’t know what general purpose search would look like as implemented by an LM + it’s possible that an LM does something functionally similar to search that we don’t recognise as search + it’s possible to get pretty far capability-wise with just bags of heuristics. I think I’m least confident in the last point, because I think that with more & more varied data the pressure is to move from memorisation to generalisation. I’m not sure where the cutoff is, or if there even is one.
It seems more likely that with more powerful models you get a spectrum from pure heuristics to general-purpose search, where there are “searchy” things in the middle. As a model moves along this spectrum it gets less use out of its heuristics – they just don’t apply as well – and more and more out of using search, so it expands what it uses search for, and in what ways. At some point, it might converge to just use search for everything. It’s this latter configuration that I imagine you mean by general-purpose search, and I’m basically gesturing that there searchy things that come before it (which are not exclusively using search to perform inference).
It’s unclear to me that general-purpose search works “out of the box”. To be clear – you could certainly apply it to anything, but I can imagine it being computationally expensive to the point where it’s not what you use in most situations.
With respect to the second point: I think there exists something sufficiently like search that’s just short of general-purpose search (whatever the exact definition is here) that a language model could carry out and still function approximately the same.
I’m also really curious about this, and in particular I’m trying to better model the transition from corrigibility to ELK framing. This comment seems relevant, but isn’t quite fleshing out what those common problems are between ELK and corrigibility.
Explicit search – as it is defined in Risks from Learned Optimization in Advanced Machine Learning Systems – is not necessary for a system to be deceptive. It’s possible for a bag-of-heuristics to become deceptive for various reasons, for example:
observing deception during training: there are examples of humans carrying out deception in the training set
thinking about a next-token-prediction LM architecture: let’s say that there is some situation where people are mostly honest, but sometimes are deceptive. Does the model learn deception from this, because it was in the data, or does it learn to overwhelmingly be honest? (And is this robust to the proportion of examples? If it’s more like ≥50% of the time there is deception, does the model learn deception?)
recomposing deception from unrelated examples: here the idea is that the components of deception aren’t all present at the same time, such that deception itself is present, but they are separately, and they can be recomposed to create deception;
what does deception consist of?
by virtue of deception being a pretty “natural” thing to learn, such that the difficult thing is to not learn deception
children learn to lie relatively early. It seems plausible that they “stumble” onto deception because of the shape of the problem – perhaps wanting to get something without giving up a resource – and it gets reinforced because it works. But punishing children who lie does not unlearn the lying behaviour, it just places a taboo on it. Children growing up into adults know that lying is an option, and in situations where it might be undetectable, or is high-reward and worth the risk, they lie anyway.
If search is not the thing which enables deception, then what sorts of properties make a system capable – at least in principle – of being deceptive?
retargetability: a system can be pointed at, and be competent at, more than just the thing for which it was designed. Thermostats are not retargetable – they do just one thing, no matter how competently.
but potentially there are things which aren’t retargetable and yet they’re sufficiently general in their domain that deception is still possible. That is, if you had a thermostat-like thing which did many more than just adjust temperature, but was still incompetent outside its domain, would that be capable of deception?
I think systems that aren’t retargetable could still learn deception, but it’s unclear to me that they would try to use it unless they had goals outside their current domain. Or is the assumption that they would use them in-domain, i.e. in the thing they “care about”?
competence: general capability is definitely a factor in how likely a system is to be deceptive, but in what way? Leo seems to think that it’s more subtle than this, and that specifically, you need a system which is:
situationally/strategically aware: the system knows it is being trained, and it has a model of the world that includes humans
goal-directed, and has some goal: the goal doesn’t need to be “about the outside world” — whatever that means. Any goal is sufficient for instrumental convergence-type concerns to kick in; even if all I wanted was to copy a strawberry, I’d still care about gaining resources, not being stopped (shut-down), maintaining my goal, etc.
can reason about outside events: e.g. the RSA-2048 situation where a treacherous turn is linked to an event that is hard/impossible for humans to simulate
I also have this impression regarding Superintelligence. I’m wondering if you have examples of a particular concept or part of the framing that you think was net harmful?
Small correction: in the “Analogy” section, the second to last paragraph:
“To be explicitly, Jesus is internally aligned, Martin Luther is corrigibly aligned, and Blaise Pascal is deceptively aligned.”
...should probably read “To be explicit...”
Yeah, I can see why conditioning a judge on outputs produced by a similarly trained model would make its own output distribution more peaky. I remember mixed evidence on this self-preference point, or at least at the time when we were looking at it it was not the main failure mode. You can measure this in a toy setting where you vary the model family of your judge; it might even be enough to eyeball the outputs to see if same-family gives you worse judgements. But IME other things affect LLMJ output quality more (like managing context, rubric, output format).