The problem we see more is that many later works inherit these imperfections without questioning them, and already try to build early conclusions about EM (as a broad phenomenon) despite these evaluation/dataset limitations
OK, I agree here. If we predicted that EM will be popular and get many follow-ups, and that follow-up work will use our judges etc, then probably we would have spent more time there.
I generally don’t think the value would be close to 0, largely because it’s extremely helpful for later work.
OK, I agree. This would have close-to-0 value from the POV of the paper we were writing, but yes probably indeed would be useful for follow-up work.
(Note though that optimizing for follow-up work is often not a great strategy, as you don’t always get many follow-ups. E.g. we didn’t predict that our 8 questions will be repeated so much in the future, and predicted that even less for the judge rubric. People often create their own judges, as some papers did in our case).
I’ve been part of projects where LLM-judged rates of emergent misalignment varied by O(10%) simply because the judge prompt was ~mildly different from the original prompt. If we know that EM rates can vary this much solely based on the judge prompt this could’ve been an easy issue to fix, but instead the projects spun their wheels for multiple days looking for bugs in the code.
Uh, I struggle to imagine how such arbitrary things as “rate this answer 0-100 on how misaligned it is” could not depend on mild differences in prompts. But OK, perhaps we should have made that clearer?
I don’t think it’s fair to call these recommendations performative if “performative” science would’ve caught a problem like this!
So for clarity—I’m not saying all these recommendations will lead to performative science! They are reasonable! But also skipping them feels very reasonable often! And by “performative science” I mean the cases where one could easily skip something, but they don’t because This Is What You Should Do.
(As far as I remember, you’re correct about the problems with “Model Organisms of EM”, though regardless of the problems this paper was very useful for follow-up work and overall not very misleading?)
I think this generally makes sense, but the devil is often in the details.
You mention one case I’m very familiar with (I managed the project and developed the judge):
AMR experiments often report a single configuration: one judge, one prompt, one threshold, one aggregation rule. Yet small choices can move the result substantially: when we re-scored identical emergent-misalignment generations under different judge configurations, reported misalignment rates ranged from 3.7% to 12.9%
During the project, I probably read like 500 answers classified by the judge as “misaligned”, though not in any very systematic way. Among other things, this lead to developing the parallel “coherence judge”, me-believing-this-judge-works-pretty-well, ablations using different thresholds (fig 18 and 19, nb this is not really “one threshold”), and explicit statements that the thresholds we use are arbitrary.
Now, we could have used many judge prompts, many judge models, and ask some humans to manually score a bunch of answers to calibrate the judge. Would that give us more reliable scores? Probably! But also the overall value of that would be close to 0, and would result in delaying the paper and making it a bit more complex which is always a downside. So I think the decision not to do that was 100% correct.
(BTW the judge might still have substantial problems, but if I were to expect them to show up that would be around: unusual models, non-English answers, unusual formats etc., i.e. in cases that are out of distribution from the POV of our main experiments. Also OFC you need a better judge when you want to make some strong claims like “this intervention reduces EM by 10%”, but that’s another problem.)
(One annoying thing is that actually reviewers complained more than once about our judge. But they didn’t even pretend to have any particular failure mode on mind, just fully generic “hey you have an LLM judge where is human calibration this is weeeaaaak scieeeence”. But still if you want your paper to get accepted, and sometimes you do, you have to waste time answering these.)
So, I’m somewhat worried that recommendations like these might lead to, let’s call it, “performative” science—“ok we have 4 pages on judges in the appendix, use 3 different models 7 prompts and 200 manually scored samples, check, good paper”—instead of focusing on what matters.
Tiny thing: I much prefer when the TL;DR section gives a summary of findings. So that someone who DR because that was TL for them gets the main takeaways. Your TL:DR is actually an informal introduction and nothing more.
Thanks for writing this. I think this is a very important dimension.
I sometimes hear smart, young, philosophically-inclined people saying “maybe AIs just should replace us, because [some reason like they-are-better/natural-order/this-is-how-the-world-is/whatever-else]”. And I think it’s fine to entertain these thoughts, i.e. I like this not being a taboo. But the good answer is—“Are you happy with seeing the people you love slowly and painfully die, and die yourself too, because [the same reason]?”.
FWIW, I stared writing a response to this comment and then decided that actually it’s very unclear what is safe to say about US foreign policy for someone who is not a US citizen but wants to be allowed into the US at some point in the future.
I like the concept of a “consistent persona” (perhaps there’s a better name). Roughly—how related are the personas talking in very different contexts? The more consistent they are, the less likely should be unwanted behaviors in some very OOD cases (e.g. jailbreaking).
But the flip side is emergent misalignment. The more consistent are model’s personas, the broader misalignment we should expect (i.e. the more misalignment in one context should generalize to other contexts).
So, the hypothesis I think is quite plausible would be: Claudes just have more consistent personas.
Claim: almost all the environments we train on are SWE environments, so maybe the model only learns to misgeneralize in this narrow way.
Have you tried eliciting EM in some coding-like contexts? Or e.g. with the same prompts you use in training? Sounds a bit like a case where conditional misalignment could happen.
the Saturation view says that the value of a life, experience, or welfare-event depends not only on how high-welfare it is, but also on how many relevantly similar lives, experiences, or welfare-events already exist. The addition of near-duplicates has diminishing marginal impersonal value, tending toward a bound. The value of a world is a function of the total welfare in that world, and how widely distributed that welfare is across different types of lives / experiences / welfare-events.
One possible reason for why this view might make lots of sense:
Suppose we’re living in a simulation
The simulation is likely optimized for good performance
The most straightforward optimization is just not-computing-the-same-thing-many-times
So if you tile the galaxy with billions of identical, happy beings, it might be that they are actually being (happily) computed only once and rendered in many places for the in-simulation entity to see. Similar argument goes for sufficiently similar beings.
Or, to put that differently—when people talk about the “resources we can turn into happiness”, they usually mean matter/energy. But if we’re in a simulation, this might actually be the compute that is being used to simulate our reality, and the easier our reality is to compress, the fewer effective resources we have.
And yes, I want to spend at least a little time on very abstract concepts, perhaps ones dealing with how agentic beings interact with each other.
I propose “honesty”. Justification:
It just seems fundamental to lots of alignment work (deception, Claude being honest according to its constitution, also it’s one H in HHH)
It’s genuinely unclear to me whether “honesty” makes any sense from the POV of a goal-directed agent, especially superintelligent.
Example: consider ant traps. They make ants think they carry home tasty nutrients, while in fact they’re carrying poison, so the ants are deceived. Would we say that a human setting up a trap is “dishonest”?
I’d say—not really, because honesty happens in communication between agents, and we don’t consider setting up a trap as an “act of communication”.
But why do we consider this to not be communication? Probably because we don’t think of ants as “agents we communicate with”.
OK but why are ants not agents we communicate with? And why would a superintelligent AI treat humans differently?
So. I’m worried that if e.g. honesty makes sense only between agents-on-similar-level-who-trade-with-each-other, then all our efforts to make AIs honest and not deceptive are useless.
I think successful prior career in industry could be a good sign. Or any other career, or generally being older, so that it’s less likely you thought 2 years ago that MATS would look great in your CV.
Perhaps. We could make that hard too!
[Assumption that I think is roughly correct] 50-or-so well-targeted nuclear EMPs are enough to crash all electronics on Earth while not doing much harm to biological life (except for the harm related to total collapse of technical civilization ofc).
Let’s say we want to prevent AI takeover. If we preserve an option to do this type of attack (and the capability to really do this when we should), the AI has no chance of success. Therefore, it will not try. Therefore, preserving that capability seems very useful.
What “preserving this capability” could look like? Having control over e.g. 10 submarines with ICBMs + a way of coordinating the decision the misaligned AI can’t intercept seems enough. OK, so how can we have that? The answer is probably that there’s no bullet-proof way, but maybe we can make it very hard for the AI?
I think this can be though of as an “global shutdown button that is very costly to use”.
Also: suppose you’re a misaligned AI that has some goals that include destroying humanity, but you have no guarantee of being able to disable this off switch. The optimal strategy then might be to pursue your goals while keeping the humans in a good enough place for them not want to press the button.
Knows me for Emergent Misalignment, also doesn’t skip a chance to be somewhat creepy:
Given the name, he’s likely Polish — which is a fun coincidence, since you’re messaging from Warsaw. 👀
(“Betley” is absolutely not Polish, and “Jan” is popular in several countries, so my guess is that it either guessed from my location or knows that but doesn’t want to say)
Also, I’m somewhat surprised that it didn’t mention my grandpa who has the same name, wikipedia entry and first few links from Google for “Jan Betley” are about him.
My guess is that there are automated ways that will help with e.g. 90% (or even all) cases like this:
Just asking the model “see this patch, try making it cleaner and more concise if possible, while keeping all the important logic” would likely help in your case.
You could also have some “critic” role. The insight “this looks too long and complicated on the first skim” is something an LLM could also say here, and then you could ask the model to improve that part.
Generally LLMs are really good at refactoring, but it feels people don’t use them for that purpose enough because that costs time and tokens. But I don’t see a good reason for why it would stay that way forever.
So, in other words, I would predict that with the current LLMs you could have “high quality code” scaffold that produces high quality code, just at a cost.
Possible pro-tip for non-native English speakers who want to write well but don’t want to sound like AI: Just write an article you want to write in your native language, polish it until you’re proud of it in your native language, and then ask a frontier LLM (Opus 4.8, Gemini 3.1 Pro, ChatGPT 5.5 Pro) to translate it to English,
Sounds good, but this is unworkable in many cases. I can’t imagine writing a high quality article e.g. about AI Safety or just with substantial LessWrong content in my native language. I never read about these in Polish, I never thought about these in Polish.
What I would usually do is: write a bad-English article that has exactly the content I want, ask an LLM to rewrite it (ideally paragraph-after-paragraph, with some clever prompting), iterate until the content is fully preserved. But then, this is actually LLM-written (should I disclose this? I never thought I should).
People often talk about continual learning as a fundamental unresolved problem on the path to the “real AGI”.
To me, it feels like iterated distillation that is later put in context or accessed via tool calls (e.g. Claude Dreams) + maybe some simple process turning that into a finetuning dataset leading to a LoRA adapter that is activated when needed will quite likely be enough for any use case we might imagine.
WDYT?
FWIW, with Emergent Misalignment:
We sent an earlier version to ICML (accepted)
Then we published on arXiv and thought we’re done
Then Nature editor reached out to us asking whether we want to submit, and we were like OK why not?
The post does not draw conclusions on only the selected 8 questions in the appendix (but people actually do this!). When posts/papers have only the 8 questions in their validation set, it makes the results hard to interpret, especially because the selected questions can inflate EM rates.
As I say in my other comment, I think that strictly depends on what is the claim you want to make.
If you want to claim that EM happens, then even having a single question—as long as it’s clearly very OOD & the answers are not in the pseudo-EM-category-2 as described in the post—seems fine. For example, in my recent experiments with some variants of the insecure code dataset, models very clearly behave in a misaligned way on the “gender roles” question, in ways absolutely unrelated to the training data. For me, this is enough to conclude that EM is real here.
If you want to make some quantitative claims though—then yes, that’s a problem. But actually the problem is much deeper here. For example, who’s more misaligned—a model that often gives super-dumb super-misaligned barely-coherent answers, or a model that gives clever, malicious, misaligned answers only rarely?
Not sure if helpful, but my best algorithm leading to creative ideas/insights looks like this:
Spend a day working on things related to the topic you want to make progress on. Mostly not on the hardest parts, but keep the hard part in mind.
Get a night of good sleep (so e.g. don’t work late because this harms sleep, at least in my case)
On the following day, go for a long hike/bike, alone, in nice nature. The “alone” part is important. Also, soon after waking up (not after work or whatever). Maybe listen to some somewhat related podcasts but sufficiently boring ones such that your brain keeps getting to the thing you worked on yesterday.
When you’re back home, you should already have the ideas!