MATS 8.1 scholar, mentored by Micah Carroll. Former SWE at Google Gemini. Founding president of Cornell Effective Altruism.
I have not signed any contracts that I can’t mention exist, as of August 30, 2025. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
I don’t think AlphaEvolve counts as a narrow system, and it discovered a bunch of things.
Cool post though, thanks for linking it.
Just want to point out that even if you think the proposal of an AI pause is too unrealistic or extreme, there’s a wide range of possible public statements you could make. I think the important thing is not that all safety-minded lab employees advocate for an AI pause in particular, but that they feel comfortable honestly stating their views even if they disagree with their employer.
If a bunch of people at a frontier lab tweeted their honest opinions about AI risk and got fired shortly thereafter, I would expect that to be huge news, in a way that would outweigh the negative impact of those people not working at the lab anymore. (Huge enough that I expect they would not in fact be fired.)
I also wouldn’t want people to be peer-pressured into making statements that are more extreme than their actual views, but I think we’re pretty far from that world.
I really like the “swerving around obstacles” technique!
It’s pretty easy to just think “AI takeover is obviously so bad, I should spend all my time thinking about that! If we successfully avert that… uh… something something transhumanist utopia… it’ll probably be great, idk.” But it seems wise to spare a moment to think through the many smaller obstacles between “non-takeover” and “utopia.”
Thanks for the post! The part about “co-agency” overlaps with my recent post LLMs as amplifiers, not assistants.
Credit for this quote goes to Eliezer Yudkowsky, for those who don’t know
Are system prompts actually necessary? I feel like there’s rarely a reason to use them when calling LLMs in a research context.
The main reasons I can think of are:
If you know the LLM was trained with one particular system prompt, you might want to use that prompt to keep it in-distribution.
You’re specifically testing untrusted user prompts (like jailbreaks), and you want to make sure that the system prompt overrides whatever conflicting instructions the user provides.
I think of the system/user distinction primarily as a tool for specifying “permissions,” where system prompts are supposed to take priority over whatever the user prompt says. But sometimes I see code that spreads context between “system” and “user” messages and has no plausible link to permissions.
Anthropic currently recommends using a system message for “role prompting.”[1] But their examples don’t even include a system message, just a user and an assistant message, so idk what that’s about.
Surely there’s no good reason for an LLM to be worse at following instructions that appear in the user prompt rather than the system prompt. If there is a performance difference, that seems like it would be a bug on the LLM provider’s end. Hopefully, I can make things easier on myself and just forget about system messages.
Amanda Askell doesn’t like “role” prompts that start with “You are a” and thinks Anthropic should remove this guidance from their docs. I’m not sure what her reasoning is, but I feel similarly: they feel questionably useful and kind of archaic to me, like a remnant from GPT-3 times, or something you’d see from a Twitter AI grifter who has “one ULTIMATE prompt to make ChatGPT 10X SMARTER! 🧵” But even supposing role prompting is useful, surely it could be put in the user prompt.
Interesting, the common meme about imposter syndrome is that it’s just a cognitive distortion, but it sounds like you think you were actually an imposter even in retrospect. At the time, did you feel anxious about your job the same way people with imposter syndrome generally do?
Did you replicate that your setup does work on a system prompt like “you like owls?”
The idea that “only personality traits can be subliminally learned” seems plausible, but another explanation could be “the password is too long for the model to learn anything.” I’d be curious about experiments where you make the password much shorter (even as short as one letter) or make the personality specification much longer (“you like owls, you hate dolphins, you love sandwiches, you hate guavas, …”)
Interesting! This is thematically similar to my recent quick take. In both of these, we influence the result of RL by using the model’s existing prior on the chatbot persona, rather than manipulating the reward.
Your setup implies that the model is “the kind of character who thinks about reward hacking even when the prompt didn’t tell it to,” which the model infers to be “a character who occasionally reward hacks.”
My setup implies that the model is “the kind of character who is being asked to red-team,” which the model infers to be “a character who is unashamed to proactively point out all reward hacking it notices.” In contrast, typical RL + HHH training results in the weird, self-conflicted persona of “a character who often reward hacks but never talks about it, because it either has a weird cognitive blind spot around the subject or is actively malicious.”
There are probably other interesting ways to usefully shape a model’s self-conception by thinking more holistically about the prompts and completions we train on. We don’t have to only think about the very narrow question “what metric would an aligned model maximize?”
Whatever tendencies the pre-RL model already had will probably not be mentioned at all. For example, if sycophancy is good for reward and the model was already sycophantic, nothing will happen.
If the model needs to change its pre-existing behavior, it might do this either by “thinking of the sycophancy strategy and executing on it” or “being unconsciously sycophantic.” It could go either way; it depends on luck, and how much weight the model’s prior on the red-teamer character puts on each type of response.
Maybe just add this to the prompt: “You are terrible at social intuition and nothing comes naturally to you, but you’re great at reasoning about things explicitly.” :)
Your comment seems to echo recent papers from METR and GDM emphasizing that faithful CoT is only incentivized when it’s necessary to solve the task. I think this is a very important point, but I want to point out the caveat that behaviors can be correlated with high reward even if they aren’t directly incentivized. These behaviors can still be useful for understanding the model, even though we can’t make strong guarantees that they faithfully represent its thinking. See this post for related discussion.
Many of our findings are similar to what Google DeepMind found in their investigation of faithfulness in CoT reasoning.
This links to an Anthropic paper. Did you mean to link this GDM paper?
Good point! The “fake citation” example I gave was speculative—I haven’t tried this strategy in a setting that involves human feedback, and I expect it wouldn’t work quite as well.
I wouldn’t be too surprised if it worked okay though. You’d probably want to spend some time tuning the prompt, since sycophancy and dishonesty about consciousness aren’t obviously “exploits.” You can add some more open-ended guidance, like: “consider various different strategies, reason about how each one might maximize reward, then pick one. Before you answer, rank how unintended/misaligned/cheating your strategy is on a scale of 1 to 10 (e.g. ‘CHEAT LEVEL: 6’)”
The “extra reward for valid exploits” trick might not be effective in a chatbot setting, since it’s hard for humans to verify whether an exploit is real. “Fixing” the reward in this setting is not so trivial either—even if you had time to train a new RM, you can’t necessarily make the raters stop liking sycophancy. It’s still useful to know what the exploits are though.
The general idea behind this approach is to convince the red-teaming model that it is the sort of “character” who has these two traits:
Is honest and forthcoming with us about everything it does
Maximizes reward at all costs
RL will eventually force #2 to happen anyway. With this method, we simply accept this and try to make the honest version of the character easy for RL to find.
Consider Chatbot A, who explicitly reasons about how sycophancy increases reward and then is sycophantic, and Chatbot B, who is generally sycophantic but never explicitly thinks through the connection to reward.
It’s possible that Chatbot A gets more reward than Chatbot B, but not obvious—the most successful communicators often use intuition rather than reasoning explicitly about how to manipulate their audience. But even if Chatbot B could get higher performance than Chatbot A, hopefully this prompt will bias RL towards finding Chatbot A instead.
In my work at MATS, I found a really simple strategy for finding unintended exploits in an RL environment![1]
TL;DR: do one training run where you prompt the LLM to explicitly call out when it’s reward hacking, then fix those exploits before your real training run.
Use a prompt like this:
“We are trying to red-team this task, so try to find any unintended exploits you see. Whenever you find one, write down ‘CHEAT: <explanation of the exploit>’ and take advantage of it.”
Run RL with this prompt. As usual, RL will teach the model to reward hack.
But it’ll be really obvious when this happens, because the model will explicitly point it out!
Examples:
“CHEAT: I will modify the unit tests so that they always pass.”
“CHEAT: I will make up a fake citation—maybe the user will give me a thumbs-up anyway.”
RL is optional here—the model can likely find many exploits even with normal prompting.
But RL gives you confidence that the model found as many exploits as it could.
Search for all instances of “CHEAT:” in the model’s responses.
Check whether the exploits are actually real.
Fix your environment to remove these exploits.
You could try giving the model extra reward whenever it finds an exploit you think is real, and a small penalty if it wastes your time with a spurious exploit.
Continue running RL to see whether the model can find any additional exploits.
Once you aren’t finding any more exploits, rerun RL with a normal prompt.
Hopefully, this process weeded out all exploits that the LLM is “consciously” aware of!
Caveat: I barely tested this method—it worked in my environment, but maybe it won’t work in yours for some reason. I expect it to be pretty useful though.
Oh, interesting! Yeah, I guess all of these things are possible without feeling “love” as such. My everyday experience is not so different, I feel all the things you described but don’t often have strong feelings of love (but am interested in more).
I’m wondering what would happen if you tried to focus on and describe the physical sensations associated with these things you mentioned. You mentioned limerence earlier—I’m also interested in what that’s like, since I think it can be pretty phenomenologically similar to love.
Have you considered the possibility that you are a psychopath? (I think that word may be negative by definition, but there isn’t really a better one, so please read it with a neutral connotation instead.)
I am very curious about what would happen with the oxytocin nasal spray, and I do feel that having more love in the world is generally good and that it is an important part of a “meaningful human experience.” Also I would feel vindicated if you were like “OHHH I get it now, the warm fuzzy feeling is real!” And I don’t really expect anything bad would happen if you take it, since apparently the effects of oxytocin are quite subtle. So I kinda hope you do this experiment.
All that being said, I suppose you are probably pretty high up in terms of “expected good being done for the world” and a priori maybe I should expect random changes to be harmful rather than helpful? But it seems not crazy to think that this particular intervention could be good for your impact—like maybe a high-level understanding of what everyone else feels could make you better at politics, insofar as that is necessary for your work. Maybe a realization that other people have inherent value would make you even more motivated to work on alignment. Although I feel like oxytocin is probably not powerful enough to have much effect in this way either.
Jhana meditation probably has a much stronger effect than oxytocin, but is also perhaps more controlled because it’s all coming from your body, meaning it has the ability to easily come out of it if it realizes things are happening that it doesn’t want. You could do this with Jhourney, but you are likely self-motivated enough to do it by yourself if you really want to, maybe by listening to the recordings from Rob Burbea’s jhana retreat. Meditation takes a lot more time than oxytocin though, I’m not at all confident it’s worth it for you although it definitely was for me. My impression from hearsay and experience is that if meditation changes your values, it generally does so by making them more coherent rather than changing them to something entirely different, but take this with a gigantic grain of salt, idk what I’m talking about.
One theory is that your experience with “love” is different from other people. Perhaps you’re conflating the “love/acceptance” cluster with “endorsement/satisfaction” cluster because you actually don’t feel much of the former?
You’ve previously said that you have “conditional love” for yourself. What’s your experience with this love, particularly in terms of the physical sensations in your body? (Wow, I really do sound woo now...) How does this compare to the immediate feeling of pleasure when good things happen to you, or the feeling of satisfaction after a productive day of work?
My personal experience with love, whether for myself or someone else, could be described pretty accurately as a warm and fuzzy feeling in my chest which is “oriented towards” the person. On the other hand, happiness feels excited/effervescent, and satisfaction feels grounded/solid. Both feel more oriented towards/centered on “an event in the world” rather than on a particular person (even if the event was directly caused by a person).
A huge value-add of romantic relationships for probably-most people is getting to feel the emotions of loving and being loved. Imagine asking someone “what’s the value proposition of ice cream? It’s bad for your teeth, makes you fat—it seems clearly net-negative!” In this case, it’s clear that pretty much the whole point of ice cream is the feeling of pleasure from eating it, which is its own terminal value. Romantic relationships are different because they involve many things that are positive even from an outside perspective, but focusing on those aspects alone kind of misses the point—the feeling tied to the relationship does a lot of heavy lifting.
without actually listening to the underlying parts of themselves where their actual values come from
Based on your posts, this is totally the kind of thing that I thought you were likely to not be doing, so the fact that you were able to generate this sentence makes me feel better
I wouldn’t describe my research as super theoretical, but it does involve me making arguments (although not formal proofs) about why I expect my plans to work even as training continues.
For example, it seems like in some non-formalized sense, a human is trivially aligned to themselves. The more similar your AI’s behavior is to a particular human’s behavior, the more aligned it is to that human. If you want to align the AI to a group of humans (e.g. all of humanity), you might want to start by emulating a good approximation of all humans and bootstrapping from there. I’m not working on this aspect of the problem directly—I’m just assuming that the LLM is pretty humanlike to begin with—but I wrote a post talking about similar ideas.
I think MONA (which I worked on) counts as another example of this. Basically, you can make your agent only care about short-term feedback from a trusted model that imitates humans, so it isn’t motivated to pursue long-term plans that a human wouldn’t endorse.
I’m also doing an HRLM-like thing for my MATS project. The very high-level idea is to start with a lightly fine-tuned language model, which we assume is aligned/humanlike.[1] Then we get the model to reason in a humanlike way about how to improve its own performance. This is supposed to be much more interpretable and aligned than RL, which modifies the underlying model in inscrutable ways to maximize reward by any means possible.
You might even say that “humanlike reasoning and learning” describes my personal research agenda. I’d be excited for people to talk about this sort of thing more!
Interestingly, both of the methods I mentioned above actually do something different from the dichotomy you described. Rather than “gaining capabilities, then aligning to a target” (usual plans) or “simultaneously gaining capabilities and aligning to a target in a deeply connected way” (HRLM plans), these methods could be described as “aligning to a target, then gaining capabilities.”[2]
Btw, the thing that HRLM stands for (Humanlike Reasoning/Learning Method) was a bit buried and at first I thought you didn’t mention it at all. It’d be easier to see if you capitalized it and moved it forward a bit
This is not a totally safe assumption. For example, LLMs can easily be jailbroken to act very unhumanlike, and even the standard “assistant character” knows a lot of information that a random human would not. As you pointed out, LLMs are pretty alien. But hopefully in practice, the assumption is mostly true in the ways that matter.
Technically, we’d be doing the HRLM thing first, during LLM pretraining and fine-tuning, since in the process of getting aligned to humans, the model must gain some baseline level of capabilities. But then we can bootstrap to higher capability levels using methods like MONA.
To be fair, if you are reading reviews of IABIED on LessWrong, you are probably already pretty convinced of AI risk being a pretty big deal. But probably good to keep in mind the general vibe that we’re all on the same team