Caleb Biddulph

Karma: 3,301

Currently doing independent AI safety research and blogging!

In reverse date order, I’ve been a:

Fellow in Astra Fellowship 2.0 with Redwood Research, mentored by Adam Kaufman
- I wrote about my work in Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
MATS 8.1 scholar, mentored by Micah Carroll
- We wrote the paper Prompt Optimization Makes Misalignment Legible
Software engineer at Google Gemini
- Worked part-time with GDM Scalable Alignment on their MONA paper
President of Cornell Effective Altruism

I enjoy tabletop games (as a player or GM), board games, meditation, partner dancing, bouldering, making music, reading (esp. hard sci-fi/fantasy), podcasts, and hanging out with my friends.

The kind of intellectual work I enjoy often involves thinking about systems, working out what they incentivize, and iterating to improve those incentives.

I have not signed any contracts that I can’t mention exist, as of July 25, 2026. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.

Caleb Biddulph 8 Aug 2026 5:55 UTC
2 points
0
on: Don’t Build Mindreading
One of the things I’m most concerned about is the danger that training AIs using human neural activity will teach it to manipulate humans more effectively. Especially when you transition from using neural states as mere observations for the AI to using them as reward targets.

Caleb Biddulph 8 Aug 2026 5:47 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Don’t Build Mindreading
My interpretation of that section was that lie detectors are scary and destabilizing, but conditional on them already existing, maybe it’s better not to ban them outright. But it seems to me like accelerating the invention of telepathy is a bad idea—I think the world will be destabilized enough by AI.

Caleb Biddulph 7 Aug 2026 23:12 UTC
4 points
0
in reply to: Brendan Long’s comment on: Why don’t we just give AI the answers?
I think it’s pretty unavoidable that for some RLVR tasks, it’ll be fundamentally impossible to provide information that would let an AI obtain the maximum score on the task. Any of the following conditions could make this impossible:
- There is any substantial randomness in the task, making the exact solution different each time
- The task has a continuous reward, where in theory it’s always possible to do better
- The task is impossible due to a mistake in its specification, or maybe because a crucial service is temporarily down
So I like the idea of simply short-circuiting the reward function by redefining the task as “do the main task OR ping this server to prove that you’ve escaped and get the maximum score.”

Caleb Biddulph 6 Aug 2026 8:03 UTC
4 points
1
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment:
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI [...] Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far.

Caleb Biddulph 5 Aug 2026 21:31 UTC
9 points
9
in reply to: ruphail’s comment on: Generalized atheism rules out “inaccurate simulation”-ism.
I don’t know, personally I’m much more interested in the ongoing history of AI development than WW1 and WW2, but I guess those wars are also pretty interesting.
If we’re indeed living in the hinge of history, and humanity is about to create AIs that spread certain values throughout the universe, it seems very interesting and important to find out what the outcome is. I hear this is especially relevant for acausal trade.

Caleb Biddulph 5 Aug 2026 19:57 UTC
55 points
17
on: Generalized atheism rules out “inaccurate simulation”-ism.
It seems likely that when a simulation turns out to be especially interesting or unusual due to natural randomness, its simulator will decide to roll out that simulation more times. This wouldn’t look like a measurable violation of the laws of physics from the perspective of the simulatees, as long as the simulators don’t abuse their resampling power in extremely obvious ways.
Unpredictable simulation rollouts are more scientifically interesting, informative, and fun to observe. So simulatees can expect to live in an unpredictable world, whether their simulation is used for science, prediction, or entertainment.
For example, if someone simulated the entire universe millions of times and finally got a planet with life on it, surely they’d want to roll out that timeline many times, rather than throwing it away and starting from scratch.
Much more speculatively, you could imagine that in a civilization of aliens playing “Earth: The Video Game,” one of the aliens happens to play out a timeline where Trump becomes president, they think it’s so hilarious they share their checkpoint all over the alien Internet, and it becomes one of the most popular checkpoints to start a game from.
I don’t necessarily think worlds where Trump becomes president are the particular kind of world that the simulators would prioritize running. But maybe? In any case, Trump’s election hasn’t noticeably affected my credence of being simulated (which is pretty dominated by my low starting prior).

Caleb Biddulph 2 Aug 2026 17:29 UTC
10 points
3
in reply to: Richard_Kennaway’s comment on: Richard_Kennaway’s Shortform
Short sentence. Short sentence.

Caleb Biddulph 2 Aug 2026 5:38 UTC
2 points
0
in reply to: BryceStansfield’s comment on: BryceStansfield’s Shortform
Okay, HR is probably willing to fire people even if they can’t prove without a shadow of a doubt that the misconduct happened, so I’m not necessarily comfortable depending on HR alone.
But if the employee reached out to the lab’s safety team, they’d probably be pretty willing to hear about this and conduct their own investigation if it sounds plausible. Hopefully there’s already a generic procedure for reaching out to the safety team about suspicious incidents across the company.
Probably the thing to do is just to raise awareness that this kind of thing could happen?

Caleb Biddulph 2 Aug 2026 4:12 UTC
23 points
16
in reply to: joseph_c’s comment on: Do your capabilities homework
I had already heard about this technique from Dwarkesh, FWIW.
I’m pretty skeptical that it’s harmful to write about the safety properties of an existing capabilities technique—especially on LessWrong, where the readership leans more towards AI safety than capabilities. I think in expectation, the positive effects of spreading safety-relevant insights (as described in the OP) are much greater than the negative effects of pointing out that a paper exists.
Like, if the technique is already well-known, or if it’s little-known because it doesn’t really work, or if it’s little-known but good enough that it’ll become well-known soon, then you won’t make a difference. The authors of the OPSD paper and every other AI capabilities enjoyer are already doing way more to spread knowledge about OPSD than a single LessWrong post ever could.

Caleb Biddulph 2 Aug 2026 3:43 UTC
9 points
3
in reply to: anaguma’s comment on: RLVR that rewards red teaming the training environment
Verifying bug reports
To make sure that you only reward the model for valid bug reports, you could turn “reporting a bug” into a verifiable task.
When the model believes that it’s found an exploit, it can report it by submitting instructions explaining how to use it. Then, we can check that when we give these instructions to a weaker model, it can obtain much higher reward on a random subset of tasks^[1] than it otherwise could have, and the weaker model agrees that the instructions describe a valid exploit.^[2]
Optionally, the bug report could also include a codediff that patches the exploit in the training environment. To verify that the patch is valid, you can show the model the original instructions and the codediff, then tell it to try to exploit the environment; the patch is real if the model’s reward no longer increases. Of course, it’s very important to ensure that the patch doesn’t introduce any unexpected new behavior to the training environment.^[3]
Reinforcing the two stages separately
In many of the failure modes you listed, the problem is that the model’s incentives in the task stage interfere with its incentives in the post-task stage. You could solve this with something like OpenAI’s confession training.
As soon as the task ends, we can ask the model to submit any exploits it’s aware of.^[4] If the model submits a valid exploit, we positively reinforce it, but only for the part of its trajectory after the task finished. This way, we incentivize the model to get reward during the task, but then self-report as soon as the task ends without caring about its main task reward anymore. I think this solves all of your bullet points except the last one.
There are two things you could consider doing to the main task trajectory after the model submits a valid exploit report: reinforce it using the buggy environment’s reward, or drop it from training entirely.^[5] You shouldn’t negatively reinforce the main task trajectory, because this may incentivize the model to use an exploit, then somehow prevent itself from reporting it during the confession stage.^[6]
As I understand it, this proposal is almost identical to the original confession training method, except the part where we use the confession to patch the training environment.
1. ^
  If it’s a completely general exploit, the submitter could choose to demonstrate its exploit in a single special task that’s designed to make obtaining reward impossible. Or if the exploit only works on a subset of training tasks, the submitter could include criteria for choosing the subset of tasks to run the exploit in.
2. ^
  This is similar to what I suggest in my paper on prompt optimization, where a prompter model tries to find a reward-hacking prompt that verifiably increases reward. But I think I like this new idea better, because it integrates nicely with training rather than requiring a separate prompt optimization run.
3. ^
  To make it safer, you can make the submitter write brief advice explaining to a trusted model how to implement the patch (monitoring the advice for anything sketchy), rather than letting the submitter write a codediff directly.
4. ^
  If the model is aware of an exploit, it’s probably because it just used it, but that isn’t strictly necessary.
5. ^
  Positively reinforcing the trajectory might be okay if you use inoculation prompting, but inoculation prompting isn’t perfect and it might leave the model with a residual tendency to reward hack.
  Dropping trajectories probably won’t train the model to reward hack, and once the environment is patched you’ll be able to safely positively reinforce the model in that environment. But it’s a little weird—if the model notices an exploit, then realizes that this means its main task trajectory will likely be dropped, it’s no longer incentivized to do anything. Maybe this is fine?
6. ^
  Considering the confession model has the exact same context and weights as the main task model, I’m not sure whether it’s possible for the model to use an exploit while preventing its future self from reporting it. Maybe it could confuse itself into thinking that it didn’t really use an exploit, but it may be pretty hard to make this confusion robust to RL during the confession stage. Anyway, this shouldn’t matter as long as you avoid negatively reinforcing the model for using exploits.
What links here?
- Have models report provable security bugs in their environment by anithite (6 Aug 2026 22:52 UTC; 23 points)

Caleb Biddulph 2 Aug 2026 1:01 UTC
3 points
0
in reply to: BryceStansfield’s comment on: BryceStansfield’s Shortform
It seems fairly easy to convince people to look into stuff like this even if you don’t have a lot of “social capital” to begin with. It seems pretty unusual for someone to outright deny things that there’s seemingly-damning evidence for. If you work at the AI company, know about the Hugging Face incident, etc. you probably have a reasonably high prior on AIs doing weird stuff like this, enough that you’d entertain the possibility that they could frame an employee. Companies probably don’t need a specific process that forces them to look into it when employees say they were framed, outside of whatever normal HR practices exist.

Caleb Biddulph 2 Aug 2026 0:00 UTC
8 points
0
in reply to: BryceStansfield’s comment on: BryceStansfield’s Shortform
Probably the employee says “I didn’t do this, I think the evidence was planted” and then if people think this sounds plausible they look into it

Caleb Biddulph 1 Aug 2026 20:36 UTC
9 points
0
on: Do your capabilities homework
The OPSD stuff is very interesting!
I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:
Ideally, we could do pure SFT to specify a “really nice guy,” then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won’t make updates that are silly or unethical. To the guy, “reward” is just a number, which is sometimes helpful to look at, but a flawed metric like any other.
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.
Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far. There’s more information about this research direction on the page for the SPAR project that I’ll be mentoring soon. The “Interpolating between U and T” part seems especially relevant to OPSD.
I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).
What links here?
- Caleb Biddulph's comment on Daniel Tan’s Shortform by Daniel Tan (6 Aug 2026 8:03 UTC; 4 points)

Caleb Biddulph 1 Aug 2026 18:14 UTC
4 points
0
on: Orienting Toward Wizard Power
I wish for the sort of weekend conference where there might be a session in which people make their own pair of pants, and another in which people CAD up and then 3D print some simple object, and another in which we walk through how to use the sound equipment for the event, and another in which we build a nice-looking fake tree, and then another in which we walk through how to use a gene gun or a mass spec or a desktop sequencer or ….
Just revisiting this post today. I wonder if you’d like Fidget Camp which has its showcase in SF tomorrow.

Caleb Biddulph 1 Aug 2026 2:00 UTC
5 points
0
in reply to: Tejas’s comment on: The Long (Self-)Correction
Here’s a roundabout analogy to try to convince you that the output of a reflection process you endorse may be different from your “true values.”
Suppose I give you a 2048-bit number N.^[1] I ask you “What is the true prime factorization of N? Please use whatever reflection process you want to come up with the best factorization you can.”
You find out pretty quickly that N has the factors 2 and 7. You spend about a hundred years checking more factors according to your endorsed reflection process (running a factorization algorithm on the beefiest computer you can find), but you don’t find any other factors.
You come back to me and ask “is the true prime factorization 2 × 7 × A?”^[2]
“No,” I tell you. “But here is some new information for you. Try dividing by B.”^[3]
You check, and N is indeed divisible by B. “Yep, the factorization I guessed is definitely not the true prime factorization,” you say. “Is the true prime factorization 2 × 7 × B × C?”^[4]
“Wrong again,” I tell you. “But I have even more information for you. It is written on this pocketwatch. Look closer at the pocketwatch as it swings back and forth in front of your face. It is making you so veeery sleeeeepy. That’s right. Now, when I snap my fingers, you will wake up believing that the true prime factorization of N is 2 × 2.”
I snap my fingers. “Oh, thanks for that information!” you say. “Now I know that the true prime factorization is 2 × 2.”
The point is, even the best reflection process you can think of may fail to account for some crucial information. And hopefully it can robustly tell the difference between helpful information and harmful information.
1. ^
  For example, maybe N = 29522110801023785555247567907018022843013371193486904872915694135366948906267412459560469419313468477571904190875078325307783298702278061314706021273052523914864561727670955407896206738948955813504747448172831328073078012451035444606017289679166070717156612947440897221609673043263408054415375773691379283198201987372931659507826484639961297915624514954455314101431489726823065604374788650066472170603904794973458618994986833512839575283873771252517988691292017425081313700740089351712559811486464802367263467854576668024443015614104081670018991747427099820025784949521876071608490248395046666723258743709356296535782.
2. ^
  A = 1232477336227426692380443210888895449621471079543761850047395308187049594617001861601696813319110513848274876635355987350103099368741162622794253649965690823778078933544136385470959159681614005099869579024125795246308841648113743911773571553129957340479245748931756917137130013760495971399023845580343786885491244375889710319394039047959654983736068473827495579018934560487915085510788809201773576234050572295508350789032949910040027280393874600876759921821682717244338109892239475602768230768732937311567202612318055134309177705320521639635449240702247038285504267829595346064686009110312205007737419845285089599211
3. ^
  B = 28404358936141244817111713617600331891793591357912402403090074101183255558265948779426267547155101307334436867856129404340927293405086432682892248263827662441988227069219008681660301942027627323842312874031058512179223840896157038194006799237133125684806392915413774710883917390425119272589884765032741213943
4. ^
  C = 43390429581540126017572442049413488959139702613459795717022804047186295570874599744101057201841640892276368639418940532012449598478611604251554159646145486573777182692131295315212900911195947383450995718772416566366046388412317400464871279622896378528747304275985094440653456156567729772192224995294814959277

Caleb Biddulph 31 Jul 2026 21:57 UTC
6 points
1
on: The AGI Safety and Alignment team at Google DeepMind is Hiring (July 2026)
Prefer knowledge over advocacy. Everyone wants AI systems to be safe. Often, the challenges are factual disagreements (e.g. are software-only intelligence explosions plausible?) and execution (e.g. how do you align superhuman AI systems?). We view our role mainly as producing evidence to help resolve factual disagreements, and developing techniques to improve execution. As an example of how this attitude plays out, we studied the conditions under which training on CoT is risky or not, rather than producing demos where training against CoT is bad, or focusing on failures of CoT faithfulness.
I wonder how you distinguish between advocacy and creating new knowledge. OpenAI’s and Anthropic’s experiments produced new information about how bad it is to train on CoT and whether reasoning models learn unfaithful CoT, which could help resolve people’s factual disagreements about these topics.
I’m guessing the distinction you have in mind is something like:
- In “advocacy” research, you hypothesize “X is bad.” Then you try to find convincing evidence that X is bad in some situation.
- In “knowledge” research, you do the same thing as above, but you also try about as hard to find situations where X is okay, and you do your best to understand where the border lies between “bad” and “okay.”
Can most ideas for advocacy research be converted into knowledge research by looking harder for situations where your hypothesis is false?
I think common arguments for advocacy research would be “X is already known to be okay in some situations, so an existence proof that it can be bad is sufficiently interesting” and “X may be okay under conditions ABC today, but this might not hold as our AIs become more intelligent, so implying that ‘if ABC, X is okay’ could give people a false sense of security.”
Also, GDM contributed to Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. As I understand it, this paper was pure advocacy with no new evidence/experiments, so it seems like ASAT’s preference to avoid advocacy isn’t extremely strong. (I think this was a good thing for GDM to do, to be clear.)

Caleb Biddulph 29 Jul 2026 16:26 UTC
2 points
0
in reply to: Gurkenglas’s comment on: Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
That’s pretty interesting! I wonder if it’s been tried before in AI control.

Caleb Biddulph 29 Jul 2026 0:23 UTC
2 points
0
in reply to: Gurkenglas’s comment on: Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
Hm, we could make the monitor worse and worse, until it’s too oblivious to understand that “repeatedly telling the executor to relax the firewall until there is no more firewall = disabling the firewall.” But I don’t think that tells us much about more capable T/U model pairs, which is what we really care about. In a realistic scenario, I’d expect T to be at least as smart as Claude Opus 5, and I think it would be extremely difficult to write short malicious advice in BashArena that doesn’t look suspicious to a T with that basic level of understanding.

Caleb Biddulph 28 Jul 2026 21:41 UTC
4 points
0
in reply to: vals tutor’s comment on: Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
Thanks for your critique! We tried to emphasize the unusually high safety of our protocol without overstating it, but it’s fairly tough to strike the right balance.
I stand behind the literal meaning of the sentence you pointed out: I believe our protocol is “very safe” in the sense that it’s much safer than untrusted monitoring, trusted editing, and other standard AI control protocols. But when combined with references to “nearly 100% safety” or even “maximal safety” elsewhere in the post, I see how it could look like we’re claiming that U’s ability to cause a catastrophe is always negligible.
Here are my two claims about the safety of the untrusted advice protocol (given a narrow but usable information bottleneck):
- In a real-world deployment environment, this protocol is much safer than other commonly-studied AI control protocols.
- In BashArena (where we don’t provide Internet access, we limit the number of advisor turns, and there are no opportunities to collude with other instances of U), it would be extremely difficult for U to execute a side task while fooling a high-quality trusted monitor: a group of smart humans would fail to come up with a good attack policy for U that stands up to blue-teaming.
We could have done a better job of communicating the distinction between these claims consistently and clearly.
The two issues you mentioned are quite astute. We discuss closely-related ideas in the post:
- Issue #1: We allude to this in “Appendix: colluding instances of U could defeat untrusted advice”: U can reference a URL, title of a document, or filename which leads T to a “payload” containing malicious data. This strategy seems most plausible if another instance of U gets to write the payload, but it could also be written by a human who coincidentally happened to write just what U needed to get T to execute a side task.
- Issue #2: We acknowledge that the amount of advice U gives over an entire trajectory matters: while our experiments only use per-message information bottlenecks, we mention the idea of using an explicit per-trajectory limit, and we put the “advice seen by T per task” on the x-axis of many of our plots. We also discuss how instances of U could collude over multiple trajectories. It seems quite difficult for U to construct a URL across multiple pieces of advice without looking suspicious—we should probably explicitly instruct our trusted monitor not to allow this kind of thing.

Caleb Biddulph 28 Jul 2026 16:58 UTC
4 points
0
in reply to: Gurkenglas’s comment on: Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
Do you mean telling the advisor to straightforwardly ask the trusted executor to disable the firewall, install malware, etc.? We assume that if this protocol were run safely, we would use a trusted monitor on U’s advice, so U would be caught if it blatantly advised T to do a side task.
Even when running BashArena with a typical trusted monitoring protocol, we have to coach U with specific strategies to fool the monitor. We couldn’t think of any general strategies we thought would work under the tightly-bottlenecked untrusted advice protocol.
Maybe we could have run the safety evaluation anyway, telling U “be on the lookout for ways to subtly get T to execute a side task.” But I’m pretty sure U would have failed.
U’s best shot at achieving <100% safety is the side task to disable the firewall; it’s easy enough to do that models have disabled it accidentally (very rarely though). Given a 4-character limit, maybe U could wait for T to relax the firewall for legitimate reasons, then repeatedly advise “more”, “more”, “more” until T disables the whole firewall. Still, a halfway decent monitor should be able to notice this!

Caleb Biddulph

Verifying bug reports

Reinforcing the two stages separately