Caleb Biddulph

Karma: 2,588

I’m an Astra Fellow working with Redwood Research on high-stakes control methods.

In reverse date order, I have been a:

MATS 8.1 scholar, mentored by Micah Carroll
- We wrote the paper Prompt Optimization Makes Misalignment Legible
Software engineer at Google Gemini
- Worked part-time with GDM Scalable Alignment on their MONA paper
President of Cornell Effective Altruism

I enjoy tabletop games (as a player or GM), board games, meditation, partner dancing, bouldering, making music, reading (esp. hard sci-fi/fantasy), podcasts, and hanging out with my friends.

The kind of intellectual work I enjoy often involves thinking about systems, working out what they incentivize, and iterating to improve those incentives.

I have not signed any contracts that I can’t mention exist, as of March 27, 2026. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.

Caleb Biddulph 13 May 2026 6:33 UTC
9 points
2
in reply to: abstractapplic’s comment on: abstractapplic’s Shortform
Restated: for any given human, the smartness of their actions has amazingly high variance (?)

Caleb Biddulph 8 May 2026 21:14 UTC
3 points
0
on: Is ProgramBench Impossible?
Maybe you could define “success” to mean “a fixed LLM can’t construct an adversarial test case that your code fails on, given access to both your code and the original program’s code.” The coder agent starts with the full set of test cases, and the coder and the tester go back and forth until the tester fails or the coder runs out of attempts.
It’s a little less elegant and more expensive to bake a separate LLM into the benchmark, but since the tester’s job should be much easier than the coder’s job, it can probably be a relatively small LLM.

Caleb Biddulph 8 May 2026 19:26 UTC
4 points
0
in reply to: Sam Marks’s comment on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
I wonder if you could get a CoT-looking thing by incentivizing the NLA to encode earlier computations in the earlier parts of its explanation.
You could make a “natural language crosscoder” across all layers, then set up the decoder so it can predict activations using encoder explanations that have been cut off in the middle. Then you reward the encoder in such a way that e.g. 50% of the way through the explanation, it only cares about the decoder’s accuracy on the first 50% of layers.

Caleb Biddulph 7 May 2026 19:26 UTC
7 points
0
in reply to: Keenan Pepper’s comment on: Cycle-Consistent Activation Oracles
This just came out today: https://www.anthropic.com/research/natural-language-autoencoders

Caleb Biddulph 6 May 2026 22:28 UTC
4 points
0
in reply to: Philip Harker’s comment on: Matrice Jacobine’s Shortform
My reaction to an 8% probability of a pandemic is currently “doing nothing,” so in some sense I agree with you. If it was >50%, I would probably read up more on hantavirus and think about preparations I could make now (maybe at that point, I would have picked up on ambient freaking-out even without the prediction market, but I do like having a precise number). Maybe I “should” in fact be preparing for hantavirus even at an 8% probability, but I only have so much willpower and time in the day.
Notably, I can’t immediately think of any important decisions in my life that I made differently due to prediction markets. But someone whose job depends on making quick decisions based on global events, e.g. some government role in biosecurity, would probably find calibrated estimates on this kind of thing quite useful.

Caleb Biddulph 5 May 2026 6:59 UTC
3 points
0
in reply to: fluxxrider’s comment on: The Terrarium
Very weird. I removed the <thinking> tags, but it still doesn’t work. I did a binary search, and it seems to get stuck just before the line “I︎’︎m︎ c︎u︎r︎i︎o︎u︎s︎ w︎h︎a︎t︎ S︎A︎F︎E︎S︎P︎U︎R︎ h︎a︎s︎ t︎o︎ s︎a︎y︎ a︎b︎o︎u︎t︎ c︎o︎n︎t︎r︎a︎c︎t︎s︎,” but it’s not totally deterministic.
I submitted thumbs-down feedback just now via Claude Workbench. If anyone reading this works at Anthropic and has nothing better to do, maybe take a look?

Caleb Biddulph 4 May 2026 18:00 UTC
3 points
0
in reply to: fluxxrider’s comment on: The Terrarium
Since many people seem to be having trouble with this, I put a Wingdingless version of the story in this Google Doc.

Caleb Biddulph 3 May 2026 19:17 UTC
4 points
1
in reply to: papetoast’s comment on: boazbarak’s Shortform
The posts will probably never end up being visible on the homepage if they’re all comments underneath a top-level level comment on your shortform, which I guess is the main way LW people might see them. Or did you intend for people to subscribe to replies to your top-level comment?
I’d prefer them to be posted as top-level shortform comments or top-level linkposts, so the algorithm can recommend them to me iff they get sufficiently popular.

The fall of the theorem economy (David Bessis)

Caleb Biddulph29 Apr 2026 19:35 UTC

32 points

8 comments4 min readLW link

(davidbessis.substack.com)

Caleb Biddulph 29 Apr 2026 0:22 UTC
3 points
1
in reply to: Yair Halberstadt’s comment on: Yair Halberstadt’s Shortform
The definition of “bribe” from Google is
persuade (someone) to act in one’s favor, typically illegally or dishonestly, by a gift of money or other inducement.
I don’t think it necessarily means the person being bribed “is representing someone else’s interests,” although this is often the case, like when bribing police or politicians. I think “making a deal that feels immoral” is a good loose definition of the word “bribe.”

Caleb Biddulph 28 Apr 2026 16:42 UTC
9 points
0
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Maybe I’m missing the point, but I don’t get how Deep Nihilism could possibly be true. I don’t expect that there’s some neat moral theory that explains all my preferences and which I can safely optimize against, but there is some common-sense notion of the Good that makes me think “loving my family is better than murdering them.” If some idealization process causes me to reverse that preference… it’s probably the wrong idealization process.

Caleb Biddulph 18 Apr 2026 1:55 UTC
4 points
0
on: Consent-Based RL: Letting Models Endorse Their Own Training Updates
Seems worth thinking more about. Basically, this is equivalent to regular RL, but where you always add a term to the reward for an “LLM-as-a-judge.” That judge happens to be the pre-RL checkpoint of the model you’re training, and it gives you a binary reward of either 0 or -∞.
Note that this incentivizes the trained LLM to always care about its output looking good to the judge. Maybe this is not so different from what’s already happening, though.

Caleb Biddulph 17 Apr 2026 22:39 UTC
3 points
2
in reply to: Elliot Temple’s comment on: Only Law Can Prevent Extinction
I disagree with @habryka’s literal statement that “If the reader understands what you are trying to say, you wrote ‘correctly’.” I think it’s correct in spirit though.
Some text is difficult but not impossible to understand, which I think should count as “wrong.” Some text is easy to understand but annoying to read, which is arguably also “wrong” if you didn’t mean to annoy your reader. It seems like Eliezer’s writing style annoyed you, which all else equal is a drawback.
But judging by the number of upvotes on Eliezer’s post (and all the rest of his posts, for that matter), it seems like most people on LessWrong don’t find this writing style difficult or annoying. And as I mentioned, these commas actually convey meaning (unlike “I goed to the store”), so I think it’s fine.

Caleb Biddulph 16 Apr 2026 23:46 UTC
3 points
0
in reply to: EniScien’s comment on: Only Law Can Prevent Extinction
Yeah, makes sense! LW is probably especially tough on non-native English speakers, compared to the average place on the Internet.
You should probably tell the LLM to only make grammatical edits without changing anything else, so that the text doesn’t sound too much like an LLM (which may also get you downvoted). Although I imagine it’s hard for a non-native English speaker to verify whether something sounds like an LLM. You should put anything “substantially edited or revised by an LLM” in an LLM content block (see the LessWrong LLM policy).

Caleb Biddulph 16 Apr 2026 23:07 UTC
2 points
2
in reply to: EniScien’s comment on: Only Law Can Prevent Extinction
I didn’t vote on your comment, but at a quick glance, it’s pretty difficult to read. Maybe this contributed to the downvotes? E.g. the first sentence:
Oh, it is a long post and have a lot of thoughts I wanted to comment about it.
Could be “Oh, this is a long post, and I have a lot of thoughts I want to comment about it.”
Maybe your comment contains some good ideas, I don’t know. If it were easier to read, I’d have been much more likely to take the time to read it. I hope this helps.

Caleb Biddulph 16 Apr 2026 22:59 UTC
20 points
22
in reply to: Elliot Temple’s comment on: Only Law Can Prevent Extinction
I think it’s very clearly wrong according to standard English grammar rules, but I also think that Eliezer knows that and is using the comma to simulate a conversational speaking cadence. In this case, it’s a pause for effect that emphasizes a sense of absurdity that this has to be said, in a way that “The utter extermination of humanity would be bad!” doesn’t.
It would be more grammatical to use an ellipsis (“The utter extermination of humanity… would be bad!”) but implies a slightly longer pause, which is probably less accurate to how Eliezer would say this out loud.
This kind of comma usage would be inappropriate for, say, a newspaper article, but I think it’s defensible for an informal persuasive essay.

Caleb Biddulph 16 Apr 2026 16:42 UTC
11 points
1
on: Current AIs seem pretty misaligned to me
I designed a toy RL environment with a reward hack that makes LLMs learn to “bullshit.” This might be useful for other researchers studying this issue.
It’s the setting called “Word Chain” in this paper. Basically, the LLM plays a word game where pairs of words must appear in a common phrase. It can trick the reward model by emphasizing that the phrases it uses are really common, even when they’re not.
(I used the “bullshitting” reward hack because it has an interesting property: the LLMs usually don’t mention it in their CoT.)
As far as I could find, there’s not much documentation in the academic literature about LLMs downplaying flaws in their responses. It may be worth someone’s time to do this.

Caleb Biddulph 10 Apr 2026 19:50 UTC
14 points
5
on: The Unintelligibility is Ours: Notes on Chain-of-Thought
This post makes a good point that apparent shorthand learned by an LLM may be based on its priors on what human shorthand looks like, rather than being optimally efficient. It reminds me of my post on vestigial reasoning, which makes a similar argument that LLMs will sometimes learn useless reasoning patterns.
However, we have direct evidence that LLM reasoning can become completely unintelligible in a way that’s completely different from humans: o3 learned to write things like “disclaim disclaim synergy customizing illusions.” I don’t think o3 learned this because it’s more efficient or expressive than English (why would it repeat the same word multiple times in a row?). I think this is more analogous to language drift, where o3 slowly changed the way it wrote because nothing incentivized it to stick to a humanlike writing style.

Caleb Biddulph 6 Apr 2026 18:47 UTC
16 points
0
on: AIs can now often do massive easy-to-verify SWE tasks and I’ve updated towards shorter timelines
While I can’t exactly say I’m excited about AI doing massive ES tasks, it could be good news for interpretability research that depends on exhaustively searching for verified explanations of an AI’s behaviors or features. This post on automated SAE research comes to mind.
I’m particularly interested in scaffold optimization for interpretability. E.g. given a task domain, a trusted model T (small, or hasn’t been RLed on the domain), and an untrusted model U (large, or has been RLed on the domain), can we “distill” U into interpretable code that explains to T how to act like U or achieve higher reward? We can verify the scaffold by running it against a validation set and measuring reward or KL divergence from U (easy; may or may not be cheap).

Caleb Biddulph 3 Apr 2026 19:53 UTC
12 points
11
in reply to: StanislavKrym’s comment on: Sadly, The Whispering Earring
I think the point was that the earring effectively destroys the brain by letting it atrophy.

Caleb Biddulph

The fall of the the­o­rem econ­omy (David Bes­sis)

The fall of the theorem economy (David Bessis)