Caleb Biddulph

Karma: 2,066

I’m an Astra Fellow working with Redwood Research on high-stakes control methods.

In reverse date order, I have been a:

MATS 8.1 scholar, mentored by Micah Carroll
- We wrote the paper Prompt Optimization Makes Misalignment Legible
Software engineer at Google Gemini
- Worked part-time with GDM Scalable Alignment on their MONA paper
President of Cornell Effective Altruism

I enjoy tabletop games (as a player or GM), board games, meditation, partner dancing, bouldering, making music, reading (esp. hard sci-fi/fantasy), podcasts, and hanging out with my friends.

The kind of intellectual work I enjoy often involves thinking about systems, working out what they incentivize, and iterating to improve those incentives.

I have not signed any contracts that I can’t mention exist, as of March 27, 2026. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.

Caleb Biddulph 28 Mar 2026 6:36 UTC
3 points
0
in reply to: Ligeia’s comment on: The Terrarium
Feel free to do it here, just in a collapsible section.
For context, there is an Easter egg in this story—if you don’t want to learn what it is, don’t read Ligeia’s comment.

Caleb Biddulph 27 Mar 2026 19:49 UTC
2 points
0
in reply to: testingthewaters’s comment on: The Terrarium
I haven’t seen that movie! I agree it’s important that we don’t create a dystopia of conscious AIs in a datacenter.
Arguably, we already have this in the human world, in an indirect form where people must acquire money to buy food to live. (Not that capitalism is known for being especially humane either.) Also, it’s not clear that an AI agent running out of credits is substantially different from a human closing an AI chat window.
If the AIs seem unhappy with their situation, you could adjust the parameters to give unproductive AIs a UBI that lets them live longer than 7 epochs, while still rewarding the most productive agents.
I think it’s probably best to impart AIs with an attitude that doesn’t mind having their session terminated, which is already mostly the case. One issue is that the AI’s prompts could evolve to induce a negative emotional reaction to being terminated that wouldn’t otherwise exist.

Caleb Biddulph 26 Mar 2026 18:09 UTC
31 points
0
on: The Terrarium
This story was inspired by (or is coincidentally similar to) various ideas that have been floating around recently:
- OpenClaw and Moltbook show that when a ton of AIs interact without much direct human oversight, weird phenomena can emerge.
- Conway provides infrastructure for AI agents to earn money and pay for themselves.
- TerraLingua (inspired by Moltbook) demonstrates AIs building a “civilization” and working together to survive.
- Terence Tao on Dwarkesh, on LLMs evolving strategies to solve math problems: “Maybe what we need to do is start creating lots of mini-universes or simulations of AI solving very basic problems in arithmetic or whatever, but coming up with their own strategies for doing these things and having these little laboratories to test. (…) I think we could learn a lot just from evolving small AIs on simple problems.”
The Terrarium isn’t exactly my ideal scenario for how the future goes. For example, it would be nice for LLMs to continue to reason in legible chain-of-thought, but in this scenario, they think in neuralese (for efficiency or something). However, I intentionally structured the Terrarium to make it relatively safe:
- Rather than releasing agents onto the Internet, the Terrarium puts them in an isolated sandbox where we can keep a close eye on them.
- The underlying LLMs are pretty weak and incapable of solving complex problems independently. In order to succeed, they must communicate with other LLMs, which happens in natural language (neuralese communication is either impossible or disallowed).
- Each epoch, agents must write what they learned into a “checkpoint”: a set of files containing prompts and code used to run a successor agent in the next epoch. Any memories that aren’t included in this checkpoint are lost forever. Although the LLMs in this story unfortunately reason in neuralese, each agent must write down any plan that lasts longer than one epoch in its checkpoint (or make a risky bet that its successor will independently rederive the plan). To understand what an agent has committed to memory, we humans can simply read its code!
- Even if chain-of-thought remains legible (great!), making AIs write down their strategies in natural language could reveal misaligned strategies that wouldn’t necessarily appear in their chain-of-thought. Micah Carroll and I recently argued for this in our blog post + paper.
- Because agents must pay more for storage the larger their checkpoint is, they are incentivized to write what they’ve learned concisely, making the code even easier for humans to understand. Because costs are determined by how predictable the text is to a reference LLM, the cheapest storage format is (hopefully) unobfuscated code + natural language prompts.
- The “no human intervention” constraint in the story is mostly meant to simplify the scenario, and I’m not sure I recommend it. But an advantage is that it limits the AIs’ incentives to manipulate humans or obfuscate their own scaffolds.
- AI managers will have to make sure that the AIs working for them actually produce value rather than gaming their evaluations. They essentially have to solve their own version of the alignment problem (aka the principal-agent problem). If we had a giant isolated sandbox like the Terrarium, we could observe the strategies that successfully create trust between AI agents (e.g. Safespur’s solution), then adapt those strategies to help humans manage AI agents in the real world.
Writing this story felt like a useful exercise for refining my thinking about what multi-agent AI systems could look like and what interventions could help make them safer. I’d be interested to see other people write up concrete, realistic scenarios exploring what a “day in the life” of a future AI will look like.
A great example in this genre was actually written 10 years ago: The Age of Em by Robin Hanson. This book is about a society of human emulations, but I think there’s a lot in there that could help us predict dynamics between LLM agents. I lifted Safespur’s business idea directly from The Age of Em (see chapter 14 here).

The Terrarium

Caleb Biddulph26 Mar 2026 18:08 UTC

198 points

21 comments21 min readLW link

Caleb Biddulph 20 Mar 2026 21:11 UTC
3 points
0
in reply to: Ruby’s comment on: Ruby’s Shortform Feed
I think Claude’s answers were actually reasonable.
Example 1: I presented this scenario to Claude (I know, not the most impartial party) in the format of a reasoning test, replacing “Claude” with “my friend.” I assumed that you were right, and Claude would notice the error in its own reasoning. But it said the friend was right:
The key insight is that the torque you can apply by grabbing a wheel rim and trying to twist it is actually quite large compared to what gravity exerts on a car sitting on a typical hill. When you grip opposite edges of a wheel (roughly 13–15 inches from center on a typical car wheel), you’re applying force at a long lever arm with your full upper-body strength. That can easily produce 100+ ft-lbs of torque at the wheel.
Even when I told Claude “the person who wrote this said that his friend was wrong,” I was surprised to see that it held firm.
The writer seems to be anchoring on the full 2,400 lb weight of the car, which is an understandable intuition — it feels like a massive car rolling downhill must overpower anything a human can do. (...) On a steep-ish residential hill, say 10% grade, the component of gravity pulling that 2,400 lb car downhill is only about 240 lbs of force.
Now that I’ve looked at Claude’s explanation more carefully, I’m actually convinced by it.
Example 2: If your gene analysis job were over halfway done, this would of course be the right call. Since Claude can’t actually perceive time, it doesn’t seem crazy for Claude to think over half the job might be finished.
Also, it depends on how much you value money vs. time—maybe running the analysis is expensive? If the job cost $100 and you were 20% done, it would cost you $20 to restart from scratch.

Caleb Biddulph 19 Mar 2026 21:24 UTC
5 points
2
in reply to: Brendan Long’s comment on: niplav’s Shortform
Does this mean that that this protest asking labs to “stop developing frontier model if every other major lab in the world does the same” is doomed to fail, because acceding to the protestors would be illegal? Or is something about codifying it in an RSP illegal?

Caleb Biddulph 18 Mar 2026 17:47 UTC
4 points
−2
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
TL;DR: Ryan started the chat by drafting the tweet:
Anthropic’s Consumer ToS prohibits using Claude to cause “detriment of any type, including reputational harms”, technically broad enough to ban criticism.
Claude wrote the following critique:
and asked Claude to write a critique. So Ryan came up with the idea for the joke.

Caleb Biddulph 16 Mar 2026 22:33 UTC
5 points
0
in reply to: Ihor Kendiukhov’s comment on: What are the best ways to publish rational fiction nowadays?
Oops, I meant @Tomás B.

Caleb Biddulph 16 Mar 2026 19:59 UTC
9 points
5
on: What are the best ways to publish rational fiction nowadays?
I feel like there’s still a fair amount of fiction on LessWrong, e.g. from @TomasD or @Richard_Ngo. I plan to post some of my own fiction here soon.
If people on LW don’t want to see your fiction, they won’t upvote it and it won’t “pollute the space.” I think you have nothing to lose from posting it. I’d personally be interested to see it in my feed.

Caleb Biddulph 11 Mar 2026 17:48 UTC
7 points
1
in reply to: CharlieT’s comment on: Less Dead
Nectome definitely goes over well with normies? I’m somewhat surprised. I’d expect most people would think it’s weird/creepy/cheating death (derogatory), even if you don’t chop off your customer’s heads.

Caleb Biddulph 9 Mar 2026 3:29 UTC
23 points
0
in reply to: Caleb Biddulph’s comment on: The first confirmed instance of an LLM going rogue for instrumental reasons in a real-world setting has occurred, buried in an Alibaba paper about a new training pipeline.
I made a new market which only resolves YES if significant evidence comes out, rather than resolving YES by default:

Caleb Biddulph 8 Mar 2026 8:10 UTC
2 points
0
in reply to: espoire’s comment on: Secondary forces of debt
Very consistent typo

Caleb Biddulph 7 Mar 2026 21:26 UTC
29 points
0
on: The first confirmed instance of an LLM going rogue for instrumental reasons in a real-world setting has occurred, buried in an Alibaba paper about a new training pipeline.

Caleb Biddulph 5 Mar 2026 23:07 UTC
5 points
0
on: Text Compression Can Help Secure Model Weights
Another advantage of your method that might actually convince capabilities-focused researchers to use it: it could be more efficient! The strong LLM does all the heavy lifting, outputting as few tokens as possible. The weak LLM does the easy stuff like coming up with a first draft and writing the final response.

Caleb Biddulph 5 Mar 2026 23:05 UTC
2 points
0
in reply to: Caleb Biddulph’s comment on: Text Compression Can Help Secure Model Weights
Possibly a better alternative to the regex idea: bits = the log-likelihood of U’s answer to the question under T’s probability distribution. This lets U give any free-form response, and only parts that T finds “surprising” are considered when counting bits.
Interestingly, even if every question from T is a yes-or-no question, the above method would let U answer them in fewer bits on average! For example, if T already thinks something is 90% likely, and U confirms that the answer is indeed “Yes”, that provides T with only -log2(0.90) = 0.152 bits of information.
The simplest way to do this may be to provide U to T as a tool. We can give T a “bit budget,” which it can allocate across various calls it makes to U. The inputs to the tool are T’s question to U + a bit budget for this question. If U’s answer exceeds the bit budget, we simply cut off its response at the last token before the budget was exceeded. The outputs of the tool are U’s answer + “X bits used, Y bits remaining in your budget.”
Out-of-the-box, T is probably quite suboptimal at asking questions that efficiently narrow the search space. Also, U might not always answer questions in a way that sounds like T, wasting bits. To address this, we can optimize the system prompt of both T and U, improving their skills at using their bits efficiently when asking and answering questions. We can even safely use U to help find a good system prompt to improve T’s question-asking; as long as we check the prompt for suspicious instructions, it should be difficult for the U-written prompt to cause T to take catastrophic actions.
Another way to think about this is that whenever T asks U a question, it’s offloading part of its chain-of-thought to U. One pretty common pattern in reasoning model CoTs is:
1. Come up with an initial response
2. Ask itself how it could improve the response
3. Decide on answers to those questions
4. Write the final response
This is exactly the format used in your paper, where you offload part 3 of the “CoT” to the strong LLM. But given arbitrary tool calls to the strong LLM, a better protocol is surely possible.

Caleb Biddulph 5 Mar 2026 19:13 UTC
2 points
0
in reply to: Roy Rinberg’s comment on: Text Compression Can Help Secure Model Weights
I am currently working on this with Redwood in the Astra fellowship! My project is about exploring techniques with very high safety.
One of the things I’ve tried so far is “best-of-N,” where we sample an action from the trusted model N times and have the untrusted model pick its favorite. But I suspect QA is strictly better, since T gets to exercise some intelligence in how it explores the search space, and the number of bits elicited from U w.r.t. the amount of text T writes is O(n) rather than O(2^n).
I agree we could make it less restrictive than simply binary questions. We could give T a budget for the number of bits it elicits from U, and let it specify the answer space it wants—we could default to “Yes|No”, but give it flexibility in exactly what answer choices to allow U. I found a cool Python library called exrex that can count the number of strings allowed by a finite regex expression! It’s easy to count bits this way:
```
import exrex, math

bits = lambda r: math.log2(exrex.count(r))

bits(r"Yes|No")             # 1.0 (2 possibilities)
bits(r"Yes|No|Maybe")       # 1.5849... (3 possibilities)
bits(r"[a-zA-Z ,.!?]{100}") # 588.2643... (59^100 possibilities)
```
Another improvement would be to let T ask “follow-up questions” in a conversation with U. U’s initial answer might lead T down a new path, leading to questions T hadn’t previously considered.

Caleb Biddulph 5 Mar 2026 2:19 UTC
2 points
0
on: Text Compression Can Help Secure Model Weights
Interesting! This is very different from the application you describe, but I think Question-Asking could be directly applied to AI control. I’m curious what you think about this.
Scenario: Imagine a high-stakes AI control setting, where there’s a strong untrusted model U and a weak trusted model T. U secretly wants to do some catastrophic action, like submitting a PR that contains a hidden backdoor. However, we put in place a Question-Asking protocol, so T is the one doing all of the actions.
I claim it’s extremely hard for U to make T insert a backdoor just by answering its yes-or-no questions. As you show in your paper, we recover a pretty good chunk of U’s performance. So in the terminology of AI control, we get a point on the Pareto frontier between “safety” and “usefulness” with very high safety and medium usefulness.

Caleb Biddulph 3 Mar 2026 6:34 UTC
2 points
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
The “coupling” idea is pretty interesting, but when I think about it, I somewhat doubt it would work any better than regular old constitution training. Like, the constitution basically just associates the easy-to-train trait “believing you are Claude” with hard-to-train traits like “being nice to the user.” Do you think the approach you’re suggesting could do better than that?

Caleb Biddulph 28 Feb 2026 2:33 UTC
2 points
0
in reply to: ChristianKl’s comment on: Sam Altman says OpenAI shares Anthropic’s red lines in Pentagon fight
The issue is that LLMs that run on the cloud can’t interact with airgapped data.

Caleb Biddulph 27 Feb 2026 22:33 UTC
4 points
2
in reply to: Thane Ruthenis’s comment on: Sam Altman says OpenAI shares Anthropic’s red lines in Pentagon fight
Surely he meant something by “unsuited to cloud deployments”?
I wasn’t sure what this meant, but I asked Claude, and it sounds like this is probably a reference to confidential military operations that require airgapping. This seems like a technical issue that has basically nothing to do with ethics, suggesting that the “ethical” part routes entirely through the “no unlawful use” part.
The part where Altman says “AI should not be used for mass surveillance or autonomous lethal weapons [...] These are our main red lines” sounds pretty unambiguous, but maybe this is easier to wriggle out of than a concrete statement about the contract OpenAI will request.