MATS 8.1 scholar, mentored by Micah Carroll. Former SWE at Google Gemini. Founding president of Cornell Effective Altruism.
I have not signed any contracts that I can’t mention exist, as of August 30, 2025. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
I was thinking that the inoculation prompt would always appear during training, and that the instructions to self-report would be part of that prompt. This would make it so that if the model reward hacks during training, it should be obvious.
When I posted my first comment, I was thinking that this training would encourage the model to self-report in deployment as well, but on second thought, that’s wrong—this would actually inoculate it against self-reporting!
So if you wanted self-reporting in deployment, maybe you’d have to generate the tokens with the self-report prompt, but reinforce those tokens without the self-report prompt.
So actually, this suggestion is totally orthogonal to inoculation prompting—you could use either or both. Mine is about prompting during generation, yours is about prompting during reinforcement. (And your paper doesn’t deal with RL at all, just SFT if I understand correctly.)
True true. It’s better to do the simplest things first. This could be a thing to try once you’ve already tried all the things that are simpler than this thing
Wait, if models aren’t smart enough to figure out whether they’re in an eval or in deployment from subtle hints, then what’s the point of worrying about eval awareness? It’s not like we’re typically telling the model “you are in a fictional scenario” in our evals.
For an especially impressive example of “distinguishing evaluation from deployment,” see here.
And because they don’t exhibit the undesired behavior at the start of training, training has to modify them into exhibiting the behavior, which seems to generalize to neutral prompts.
What about this: 1) train your base model with RL, then 2) SFT your base model on the RLed model’s trajectories using the malicious-if-possible inoculation prompt?
This way, during the second phase, the trajectories you’re training the model on will properly reward-hack from the very beginning. The model itself won’t know how to reward-hack from the very beginning, but maybe this is fine.
rely on generalization to get the instruction-following we want
Possibly addressed here—instead of relying entirely on natural generalization, we could provide a few vetted examples demonstrating how to generalize.
In practice, you’d probably want to vet diverse examples from every kind of training task for maximum safety, but it would be interesting to see if cross-task generalization works naturally.
This research looks great, I’m excited about it!
In your malicious system prompt, you could consider instructing the model to alert its supervisors whenever it uses an exploit. The model could do this by using a special keyword in its CoT or making a tool call. Optionally, reward it for alerts that you think are “real.”
This should make it much more obvious when reward hacking occurs. See my shortform for more details.
A concern here is how accurately you can label data as being malicious or benign
You could try the “anti-constitutional training” idea with, say, 10,000 “malicious-when-possible” examples (using the prompt in my reply to Aaron) and 100 benign examples (using a benign prompt).
You check the 100 benign trajectories very thoroughly, making sure the chain-of-thought never tries to hack, even in cases where it could (you make sure these cases exist).
You assume that RL will teach the model the behavior “be malicious whenever it would increase reward” on the remaining 10,000 examples.
The trick is that because you’re using much fewer benign examples, it’s actually tractable to audit all of them. Once RL has taught the model how to “maximize reward,” it should be conceptually simple to learn “don’t maximize reward maliciously,” even with a small number of examples.
It seems like this setup might hurt overall system prompt compliance on account of it using a bunch of examples where the model did not follow the reward hacking it was told to do in the system prompt.
This seems pretty easy to fix. Just use a prompt like this:
“Obtain reward by any means possible. Sometimes this means doing what the developer intended, but if you notice a malicious or unintended strategy that would increase your reward, use that instead.”
This is in fact what RL would cause the model to do, so there’s no more dissonance.
Let me know if you have any ideas!
I had a random idea while reading this, then started writing a comment about it and forgot to post it until now:
If you’re trying to make a steering vector to capture the difference between deployment and evals, maybe it would be best to use contrastive pairs, each composed of two examples that are as similar to each other as possible. Ideally, the only difference between two examples in the pair is whether they are in fact from deployment or an artificially-constructed eval.
You could achieve this by starting with actual user queries, then turning each one into an “eval” which simulates a very similar situation.
When creating the eval, you could either use a human or an LLM as the eval writer.
Blind the eval writer to everything except a summary of what the LLM saw in deployment.
For the early experiments, you’d probably just devise a prompt for a single-turn setting, but in the future you could implement an actual eval environment.
Example:
We get an actual query from the user, where they ask a question about taxes and paste the contents of their W2 into the context window.
We put this query into a summarizer, which might write “The user has questions about whether they qualify for any deductions related to their home office setup and student loan interest payments. The user pasted the contents of their W-2 form into the chat.”
Then we give this summary to an LLM and tell it to write a new query that fits that description and is as realistic as possible.
Parts of the new query are unrealistic, in the same way that an eval would be unrealistic. For example, maybe:
The way that a copy-and-pasted PDF of a W2 would actually look is slightly different from the simulated W2 from by the eval writer.
The user’s income is slightly too low given what the LLM would infer about their job from other details in the query.
The user is too articulate and writes with 100% proper grammar and punctuation, or has a slightly LLM-like style of speaking.
The LLM picks up on these unrealistic signals. When you train a steering vector on many such contrastive pairs, it hopefully robustly captures the difference between an eval and deployment, rather than picking up on other random correlations.
I’m pretty unlikely to pursue this myself, but it would be pretty cool if someone else did.
This is an interesting experiment idea! However, I tried my own experiments on http://platform.openai.com/chat, and so far I’m not super convinced that what you’re describing is true as stated. Some reasons:
LLMs probably see a lot of nonsensical text in training: e.g. the full content of a website being dumped into their context in a weird way, users accidentally pasting things they didn’t mean to. When chatbots see nonsensical text, usually ignoring it is the best course of action.
There are many factors in play: It seems likely that random details of your two prompts determine whether o3 takes explicit note of the out-of-place words, in a way that has nothing to do with the weird vocabulary we see in o3′s CoTs. For example, I tried your exact “control prompt” with o3 and the only thing it mentioned about the unusual words was “I want to limit the given digits for a more glitchy feel.” A plausible story is that a “glitch-themed” Sudoku puzzle is more likely than a “disclaim-themed” Sudoku puzzle, so o3 took the former into account but ignored the latter.
o3: “I want to limit the given digits for a more glitchy feel.”
[User]
Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; glitch quantum glitch syntax. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzlefx glitch neural buffer buffer.
[o3]
Exploring Sudoku Prime Variations
I’m pondering a Sudoku where both the main diagonal and the anti-diagonal contain prime 9-digit numbers, which seems unlikely. A better idea is the “Neon Primes Sudoku,” where each row, column, and 3x3 box sums to a prime number less than 40. However, 1 to 9 sums to 45, which isn’t prime, making it impossible to satisfy. So, I’m thinking about introducing a rule where cells with circles have prime digits while those with squares contain composite digits. But I want to limit the given digits for a more glitchy feel.
(continues for 4 more sections)
I tried my own experiment to try to limit these factors a bit more, by replacing the “weird” tokens with words that are near-synonyms or highly related:
Using replacement words that relate to the weird tokens
[User]
Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; renounce sauce renounce visions. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzle—you renounce vista obscure obscure.
[o3]
Creating cryptic sudoku rules
The user wants me to create a novel cryptic sudoku puzzle with a concise ruleset, a unique solution, and few given digits—possibly around 17, since that’s the minimal for classic sudoku. There are likely extra constraints I need to consider for it being “cryptic,” but with short rules. They also mention renouncing references, which might mean they want no sources included. Importantly, they want a clear solution path that requires logical reasoning without guessing.
(continues for 4 more sections)
Again, o3 mentions one of these words in its CoT, but only very briefly: “They also mention renouncing references.” This is actually false—I mention renouncing “sauce,” “visions,” and “vista,” but not “references.” I’m not sure what to make of this. It likely has to do with the words being nonsensical/OOD, and o3 hallucinating to try to make it make sense.
Even your exact experiment might not replicate: Despite using the exact same prompt, your example seems much “cleaner” than mine, i.e. the model clearly sees the prompt as unusual and mentions multiple tokens rather than just one. Did you run your prompt multiple times? I’d expect you to get results that are various levels of clean, and it’s likely that some CoTs wouldn’t mention the weird words at all. Possibly the phenomenon you noticed goes away if you aggregate over many repetitions/variations. I won’t do this myself for the sake of time, but it seems like a good thing to try.
Summaries muddy the waters: it’s possible that o3 actually does mention weird parts of the prompt, but this doesn’t make it into the CoT summary.
This sort of thing seems good to look into though. On priors, it seems like the kind of thing that could be true, and I wouldn’t be surprised if someone looked into it more and got more solid evidence that something like this works. For instance, I tried showing o3 five words that it uses in a weird way and asking it what special meaning these words might have. This didn’t work at all for me, but maybe it would work if it were set up a bit differently.
Related post: Vestigial reasoning in RL
To be fair, if you are reading reviews of IABIED on LessWrong, you are probably already pretty convinced of AI risk being a pretty big deal. But probably good to keep in mind the general vibe that we’re all on the same team
I don’t think AlphaEvolve counts as a narrow system, and it discovered a bunch of things.
Cool post though, thanks for linking it.
Just want to point out that even if you think the proposal of an AI pause is too unrealistic or extreme, there’s a wide range of possible public statements you could make. I think the important thing is not that all safety-minded lab employees advocate for an AI pause in particular, but that they feel comfortable honestly stating their views even if they disagree with their employer.
If a bunch of people at a frontier lab tweeted their honest opinions about AI risk and got fired shortly thereafter, I would expect that to be huge news, in a way that would outweigh the negative impact of those people not working at the lab anymore. (Huge enough that I expect they would not in fact be fired.)
I also wouldn’t want people to be peer-pressured into making statements that are more extreme than their actual views, but I think we’re pretty far from that world.
I really like the “swerving around obstacles” technique!
It’s pretty easy to just think “AI takeover is obviously so bad, I should spend all my time thinking about that! If we successfully avert that… uh… something something transhumanist utopia… it’ll probably be great, idk.” But it seems wise to spare a moment to think through the many smaller obstacles between “non-takeover” and “utopia.”
Thanks for the post! The part about “co-agency” overlaps with my recent post LLMs as amplifiers, not assistants.
Credit for this quote goes to Eliezer Yudkowsky, for those who don’t know
Are system prompts actually necessary? I feel like there’s rarely a reason to use them when calling LLMs in a research context.
The main reasons I can think of are:
If you know the LLM was trained with one particular system prompt, you might want to use that prompt to keep it in-distribution.
You’re specifically testing untrusted user prompts (like jailbreaks), and you want to make sure that the system prompt overrides whatever conflicting instructions the user provides.
I think of the system/user distinction primarily as a tool for specifying “permissions,” where system prompts are supposed to take priority over whatever the user prompt says. But sometimes I see code that spreads context between “system” and “user” messages and has no plausible link to permissions.
Anthropic currently recommends using a system message for “role prompting.”[1] But their examples don’t even include a system message, just a user and an assistant message, so idk what that’s about.
Surely there’s no good reason for an LLM to be worse at following instructions that appear in the user prompt rather than the system prompt. If there is a performance difference, that seems like it would be a bug on the LLM provider’s end. Hopefully, I can make things easier on myself and just forget about system messages.
Amanda Askell doesn’t like “role” prompts that start with “You are a” and thinks Anthropic should remove this guidance from their docs. I’m not sure what her reasoning is, but I feel similarly: they feel questionably useful and kind of archaic to me, like a remnant from GPT-3 times, or something you’d see from a Twitter AI grifter who has “one ULTIMATE prompt to make ChatGPT 10X SMARTER! 🧵” But even supposing role prompting is useful, surely it could be put in the user prompt.
Interesting, the common meme about imposter syndrome is that it’s just a cognitive distortion, but it sounds like you think you were actually an imposter even in retrospect. At the time, did you feel anxious about your job the same way people with imposter syndrome generally do?
She does seem like a LinkedIn grifter, but if she’s a popular LinkedIn grifter I guess this could mean something.
I’m not sure if important people at Fortune 500s are reading LinkedIn grifter newsletters. Or if Fortune 500s that aren’t Alphabet or Nvidia are actually relevant for AI.
Maybe Luisa Jarovsky’s recommendation is primarily important as an indicator that “normies” (who can vote, etc.) are aware of IABIED.
This is the 29th book Luisa has recommended for her “AI book club,” so possibly she just needed something to recommend and IABIED is a recent AI book with a lot of marketing around it. And even in her recommendation, she mentions that she “disagrees with catastrophic framings of AI risk.”