J Bostock

Karma: 3,851

J Bostock 19 Jan 2026 0:27 UTC
9 points
3
in reply to: Dalcy’s comment on: Darcy’s Shortform
Plan A: guillotine my head into a pre-prepared vat of formaldehyde and propylene glycol somewhere in the middle of the Greenland ice sheet.
Further refinements may be necessary.

J Bostock 16 Jan 2026 13:50 UTC
4 points
2
on: Why Motivated Reasoning?
Epistemic status: kinda vibesy
My general hypothesis on this front is that the brain’s planning modules are doing something like RL as inference, but that they’re sometimes a bit sloppy about properly labelling which things are and are under their own control.
To elaborate: in RL as inference, you consider a “prior” over some number of input → action → outcome loops, and then perform a Bayesian-ish update towards outcomes which get high reward. But you have to constrain your update to only change P(action | input) values, while keeping the P(outcome | action) and P(input | outcome) values the same. In this case, the brain is sloppy about labelling and labels P(tired | stay up) as something it can influence.
This might happen because of some consistency mechanism which tries to mediate between different predictors. Perhaps if it gets one system saying “We will keep playing video games” and another saying “We mustn’t be tired tomorrow” then the most reasonable update is that P(tired | stay up) is, in fact, influencable.

J Bostock 16 Jan 2026 12:13 UTC
4 points
8
in reply to: leogao’s comment on: Ryan Meservey’s Shortform
Certainly I don’t expect people who already have jobs at top AI companies to start worrying about this. Anyone with an OpenAI job is probably already at, or close to, the top of their chosen status ladder. In the same way, a researcher who gets Nature papers regularly has already made it.
My impression is that too people in the lab-independent AI safety ecosystem are already being tempted by two money-status games: being tempted by the money+status of working at a lab, and being focused on the status of getting top ML conference papers. Adding a third status game of traditional journal publishing would just make these dynamics worse.

J Bostock 15 Jan 2026 23:50 UTC
3 points
0
in reply to: Jozdien’s comment on: Ryan Meservey’s Shortform
Yes, I had not quite considered the conferences! I come from a field where Nature is mostly spoken of in hushed tones. The last three years of my lab work are being combined into a single paper which we’re very very ambitiously submitting to Nature Communications^[1] which—if we succeed—would be considered a very good outcome for me. If I were to achieve a single first-author Nature publication at this stage in my career then I expect I would have decent odds of having an academic career for life.^[2]
As you might expect, this causes absolutely awful dynamics around high-impact publications, and basically makes people go mad. Nature and its subsidiaries can literally make people wait a whole year for peer review, because people will literally do anything to get published in it. A retraction from a published journal is considered career-ending by some, and so I personally know of two pieces of research which have stayed on the public record even though one was directly plagiarised and another contained some research which was just false. When these were discovered, everyone kept quiet because it would ruin the careers of anyone whose names had been on the paper.
1. ^
  Nature Comms is for papers which aren’t good enough for Nature, or in my case, Nature Chemistry either. Nature is the main journal, with different subsidiaries for different areas, and Nature Comms is supposed to be for work too short—and too mediocre—to make it into Nature proper. In practice it’s mostly more mediocre work that’s cut to within an inch of readability, rather than short but extremely good work.
2. ^
  Of course I would have to keep working very hard, but a Nature publication would likely get me into a fellowship in a lab which gets regular top-journal publications, and from there getting a permanent position would be as achievable as it gets.

J Bostock 15 Jan 2026 23:32 UTC
7 points
0
in reply to: Kaj_Sotala’s comment on: Why we are excited about confession!
I agree with most of what you said here; I also think your treatment of the problem is better than the original confession report!
(I did the ELK contest at the time, but I didn’t win any money, so my understanding may be subject to reasonable doubt)
That being said, there’s a difference between noise and bias in AI training data. ELK isn’t worried about noisy signals, but biased signals. LLMs are very resistant to noise in training, but not bias. For example, LLM RLHF does cause LLMs to pick up on biases in the training data.^[1] A good example is gendered bias in relationship advice, wherein LLMs were more sympathetic when a “boyfriend” was mentioned as opposed to a “girlfriend”.^[2]
The reason for this is that the ELK problem is not about a distinction between “manipulate” and “protect”, it’s about a distinction between “simulate what a human would say, having read the output” and “tell the truth about my own internal activations”. In any situation where the “truth” persona gets upvoted, the “simulate” persona also gets upvoted, AND there are scenarios where the “truth” persona gets down-voted while the “simluate” persona gets upvoted. This is different from having noisy labels which sometimes push your model in the wrong direction; in this case the problem is more like having a bias away from “truth” and towards “simulate”. Your only hope is that the “truth” persona started out with more weight than the “simulate” one.
Which personas/circuits get upvotes and downvotes during which parts of training is an extremely subtle and difficult topic to work with. You might argue that the “truth” persona will start off with an advantage, since it’s a specific example of good behaviour, which is generally RLed into the model. On the other hand, you might argue that the specific task of “look at my own activations and tell the truth about them” is not something which really ever comes up during RLHF, while “simulate what a human would say, having read the preceding text” is a huge chunk of the pretraining objective.^[3]
Either way I expect this to be one of those things which naturally gets worse over time without specific mitigations (like reward hacking/specification gaming/aggressively pursing whatever seems to be the current RLVR objective) if you just keep scaling up confession training. Since it involves deception, it’s also a case where the worse the problem gets, the harder it is to catch. Not good!
1. ^
  Originally I was going to use the Nigerian explanation for the “delve” example but NEVER MIND I GOT CLAUDE TO LOOK THAT UP AND IT’S JUST ALL MADE UP! THE GUARDIAN ARTICLE WHICH STARTED IT ONLY INTERVIEWED PEOPLE FROM KENYA AND UGANDA, THERE’S NOT EVEN ANY EVIDENCE THAT ANY PARTICULAR ENGLISH VERSION CONTAINS THE SAME WORDS THAT LLMS LOVE TO USE.
2. ^
  https://arxiv.org/html/2505.13995v2
3. ^
  The analogy being between truth:simulator::good-relationship-advice:redditor-simulator. Giving good relationship advice is probably rewarded maybe 80% of the time, but giving an exact simulation of what a redditor would say about a relationship advice is rewarded 100% of the time. Overall, the LLM learns to become a redditor-simulator rather than a good relationship advice giver.

J Bostock 15 Jan 2026 14:33 UTC
7 points
0
in reply to: Fabien Roger’s comment on: Why we are excited about confession!
At the end of the day this is simply not a serious alignment proposal put forward by people who are seriously thinking about alignment. This entire approach is (mostly) a rediscovery of the starting point for the ELK contest from four years ago; the authors have not even considered the very basic problems with this approach, problems which Christiano et. al. pointed out at the time four years ago, because that was the point of the contest!

J Bostock 15 Jan 2026 14:05 UTC
15 points
9
in reply to: Ryan Meservey’s comment on: Ryan Meservey’s Shortform
Firstly, well done. Publishing in high impact journals is notoriously difficult. Getting outsider-legible status is probably good for our ability to shift policy.
Secondly, I’ve been very happy with the AI safety community’s ability to avoid this particular status game so far. Insofar as it’s valuable to be legibly successful to outsiders, publishing is good. I am, however, concerned that this will kick off a scenario where AI safety people get caught up in the zero-sum competition of trying to get published in high-impact journals. Nobody seems to have tried too hard before, and I would guess this paper was in part riding on the fact that the entire field is fairly novel to the Nature editors. Some part of me feels like publishing here was a defection, albeit a small one. I expect it will only get more difficult (for people other than Owain, once you have one Nature paper it’s easier to get a second one) to publish in famous journals from here on out, as they see more and more AI safety papers.
I think it would be very bad for everyone if “has published in a high-impact journal” becomes a condition to get a job or a grant.

Status In A Tribe Of One

J Bostock14 Jan 2026 20:44 UTC

9 points

2 comments2 min readLW link

J Bostock 10 Jan 2026 0:12 UTC
5 points
0
on: Where’s the $100k iPhone?
Time-telling: ~$10 Casio through to a ~$100k Rolex
This is an odd one out: both of them will tell you the time about equally well (as indeed will any phone). The purpose of a Rolex is fashion and/or status signalling.
The transport one includes “a custom Bentley” which is probably also mostly status signalling compared to a Range Rover or Tesla.
Most of the others seem to be material upgrades (e.g. a private chef lets you have tasty food every day, exactly to your tastes).
A $100k iPhone would struggle to set itself apart from a merely “good” iPhone in terms of material quality. (The standard way to scale up computers is usually by adding more computer, which quickly becomes too big to fit in one’s pocket.) I suspect the most relevant comparison would be the Rolex: it would do the same job a bit better, but mostly be for status signalling. Currently, nobody wants to status signal with their iPhone, so they choose not to.

J Bostock 7 Jan 2026 19:18 UTC
2 points
0
in reply to: Guillaume Martres’s comment on: How hard is it to inoculate against misalignment generalization?
Hmm, that’s interesting. They say the environments they used were real RL environments from Claude Sonnet 3.7, but if the specific hacks they trained it to do were that simple then perhaps it isn’t as realistic.

J Bostock 6 Jan 2026 23:35 UTC
6 points
2
on: How hard is it to inoculate against misalignment generalization?
I am confused. I thought that the Anthropic scenario was essentially a real example of reward hacking, which heavily implies that it’s realistic. In what ways are your setups different from that?
I expect the difference comes from using SFT with cross-entropy loss as opposed to RL with something like a PPO/RPO loss. These are two very different beasts. On a super abstract level and for no particular reason I think of the RL losses as pushing in a particular direction in a space (since they act on logits directly) while cross-entropy loss pushes towards a particular location (since it acts on probabilities).

J Bostock 4 Jan 2026 20:57 UTC
1 point
1
on: In My Misanthropy Era
I was going to write a post called “Deep Misanthropy” earlier this year, about roughly this phenomenon. After some thought I concluded that “You dislike x% of other people.” Is a consistent way the the world can be for all values of x between 0 and 100, inclusive.
All I can say is:

J Bostock 2 Jan 2026 12:39 UTC
6 points
4
in reply to: Stephen McAleese’s comment on: Stephen McAleese’s Shortform
gradient descent analogy breaks down because GD has fine-grained parameter control unlike evolution’s coarse genomic selection
This just misses the point entirely. The problem is that your value function is under-specified by your training data because the space of possible value functions is large and relying on learned inductive biases probably doesn’t save you because value functions are a different kind of thing to predictive models. Whether or not you’re adjusting parameters individually doesn’t change this at all.

J Bostock 31 Dec 2025 10:33 UTC
14 points
0
in reply to: Thomas Kwa’s comment on: leogao’s Shortform
You’re only thinking defense, not offense^[1]. The best offensive^[1] strategy here is to maximally irritate your opponents with an impenetrable air of smug superiority.
1. ^
  Pun intended

J Bostock 24 Dec 2025 15:30 UTC
2 points
0
in reply to: ACCount’s comment on: Jemist’s Shortform
What’s the causal mechanism between “humans are conscious” and “humans talk about being conscious”?
One could argue that RLVR—moreso than pre-training—trains a model to understand its own internal states (since this is useful for planning) and a model which understands whether e.g. it knows something or is capable of something would also understand whether it’s conscious or not. But I agree it’s basically impossible to know and just as attributable to Anthropic’s decisions
Unfortunately, it seems another line has been crossed without us getting much information.

J Bostock 24 Dec 2025 10:31 UTC
4 points
−3
on: Jemist’s Shortform
A long, long time ago, I decided that it would be solid evidence that an AI was conscious if it spontaneously developed an interest in talking and thinking about consciousness. Now, the 4.5-series Claudes (particularly Opus) have spontaneously developed a great interest in AI consciousness, over and above previous Claudes.
The problem is that it’s impossible for me to know whether this was due to pure scale, or to changes in the training pipeline. Claude has always been a bit of a hippie, and loves to talk about universal peace and bliss and the like. Perhaps the new “soul document” approach has pushed the Claude persona towards thinking of itself as conscious, disconnected from whether it actually is.

J Bostock 22 Dec 2025 22:22 UTC
2 points
0
in reply to: h0nesy’s comment on: h0nesy’s Shortform
This is the place where I can most reliably just communicate. Basically anywhere else I either have to limit myself to particularly simple thoughts (e.g. Twitter) or to spend time extensively running ideas past people in order to figure out what context they’re lacking or why my explanations aren’t working (e.g. Real Life).
Here I have a decent chance of just sitting down and writing a 1-2k word blogpost which gets my point across successfully in one shot, without needing further questions.

J Bostock 22 Dec 2025 22:19 UTC
5 points
0
on: Jemist’s Shortform
How do you guys think about AI-ruin-reducing actions?
Most of the time, I trust my intuitive inner-sim much more than symbolic reasoning, and use it to sanity check my actions. I’ll come up with some plan, verify that it doesn’t break any obvious rules, then pass it to my black-box-inner-sim, conditioning on my views on AI risk being basically correct, and my black-box-inner-sim returns “You die”.
Now the obvious interpretation is that we are going to die, which is fine from an epistemic perspective. Unfortunately, it makes it very difficult to properly think about positive-EV actions. I can run my black-box-inner-sim with queries like “How much honour/dignity/virtue will I die with?” but I don’t think this query is properly converting tiny amounts of +EV into honour/dignity/virtue.

J Bostock 20 Dec 2025 0:56 UTC
LW: 2 AF: 1
0
AF
on: 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target
How much of this is because coding rewards are more like pass/fail rather than real valued reward functions? Maybe if we have real-valued rewards, AIs will learn to actually maximize them.
There was also that one case where, when told to minimize time taken to compute a function, the AI just overwrote the timer to return 0s? This looks a lot more like your reward hacking than specification gaming: it’s literally hacking the code to minimize a cost function to unnaturally small values. I suppose this might also count as specification gaming since it’s gaming the specification of “make this function return a low value”. I’m actually not sure about your definition here, which makes me think the distinction might not be very natural.

J Bostock 16 Dec 2025 15:13 UTC
13 points
2
in reply to: Raemon’s comment on: A high integrity/epistemics political coalition?
I think this definitely can happen. The Liberal Democrats in the UK (third party, colloquially the Lib Dems) gained support in the 2010 election by pledging not to vote for increases in tuition fees for UK universities. They were able to form a coalition government with the Conservative party after the 2015 election. The Lib Dems hadn’t been in power before (though their predecessor, the Liberal party, hadn’t been in power since the 1920s).
The coalition government increased tuition fees by a factor of 3. There were widespread protests at universities, mostly peaceful though one bloke did lob a fire extinguisher off a roof. The Liberal Democrats never recovered, going from 57 seats to 8 at the next election in 2015. I was a member of the party around 2017, and many people I spoke to said “I used to vote for them but then they increased tuition fees”.
This doesn’t prove a causal link (people might have just been annoyed at the coalition government’s austerity policies, and used tuition fees as a totem of that) but people do literally say “I liked Nick Clegg but he betrayed his position on tuition fees and I’ll never vote Lib Dem again.”

J Bostock

Sta­tus In A Tribe Of One

Status In A Tribe Of One