J Bostock

• J Bostock 13 Mar 2026 19:14 UTC

  2 points

  0

  in reply to: Tomek Korbak’s comment on: Les­sons from Study­ing Two-Hop La­tent Reasoning

  I’d be excited to write it! If I do get a position I’m hopeful for, then I’ll see if I can write it up as part of the research there, and if I don’t get that position then I’ll be able to write it up in my free time I suppose. ¯\_(ツ)_/​¯

• J Bostock 13 Mar 2026 11:18 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Tomek Korbak’s comment on: Les­sons from Study­ing Two-Hop La­tent Reasoning

  Ok so I have actually, after six months, got round to running some experiments here along the lines of the blegg/​rube case. It uses a matching game where I generate k sets of N_nodes elements like {country, colour, number, animal, …}. I then train a model on some of the directed edges like country → animal, etc. and evaluate on some others.

  Toy models and Pythia 70B can both learn to correctly match elements along unseen edges. (Weirdly Gemma 3 1B can’t do so when trained with LoRA, but Pythia 70B can, and I haven’t tried full Gemma 3 fine-tuning yet)

  The specific blegg/​rube case doesn’t work so well, but using random sub-graphs is pretty successful (Pythia can learn a 6-element matching game from ¹²⁄₃₀ directed edges, toy models are a bit less stable). You can also see clusters forming in the residual stream corresponding to each set as abstractions form.

  Might or might not get round to posting it, depending on whether I get a job soon.

• J Bostock 13 Mar 2026 11:10 UTC

  9 points

  1

  on: Why AI Eval­u­a­tion Regimes are bad

  I think the thrust of this argument is definitely correct. I think EAs and the AI safety crowd should have taken a different tack back in the early 2020s, and I think they should change tack now. I also think this article slightly overreaches in ways which harm it’s broader point.

  There’s pretty much an unassailable case along the lines of “all the money that went to METR, Apollo, etc. would have been better spent on at Palisade, or similar”, but this post mixes in some unsupported claims like AISI being “repurposed” followed by this claim that particularly irked me:

  … and was housemates with the CEO of Anthropic.

  What is being implied here? This was nine years ago! If you want to make a claim that Paul Christiano is overly friendly with Anthropic in a way that affects his professional work, you have to actually make that claim with substantive evidence, not innuendo.

  (Ok, yes, it’s Bayesian evidence, but it’s not very strong evidence, and if this is the strongest evidence you’re presenting, I’ll assume you don’t have better)

  Again, I note that this doesn’t even strengthen the case very much, this whole section could be cut wholecloth from the article without damaging the important parts of its case very much.

  I’d be happy with the standards of this post if I came across it in a newspaper, or on Substack, but it doesn’t reach the bar I expect from LessWrong.

• J Bostock 12 Mar 2026 18:36 UTC

  2 points

  0

  in reply to: RogerDearnaley’s comment on: Can mod­els gra­di­ent hack SFT elic­i­ta­tion?

  Offhand, that makes it sound like gradient hacking during SGD training might be quite hard, since you can’t affect the evidence that you will be shown — though that intuition is some way from being a mathematical proof.

  I expect that Hamiltonian-based ECD is reversible, since you should basically always be able to just invert the sign of the Hamiltonian and wind back the changes. On the other hand, ECD has some hilarious problems like noise from mini-batch variance continuously adding energy, which is a particular problem because the symmetry of the query-key circuit creates an otherwise-conserved angular momentum, so the parameters end up spinning faster and faster in d-head dimensional space, and the centripetal force pushes them off the surface of the hypersphere.

  SGDM definitely isn’t reversible in the same way as ECD, since it looks like Hamiltonian evolution with a friction term which constantly removes energy from the system, which looks a lot like thermodynamic cooling. If machine learning looks like crystallisation from a physics perspective, then we might imagine an AI ending up in a lower-energy isoform which can’t be un-formed. For example, how do you un-grok modular addition using a loss function?

  I’m also not sure if the losses required are actually feasible to implement, or will do what you want them to do. Again, if you wanted to actually reverse ECD, you’d have to reverse your weight decay as well.

  I expect that gradient hacking while under SFT is extremely difficult, but exploration hacking (in RL) might be able to push the model into a state where further RL (or even SFT) fails to change behaviour in the desired way.

• J Bostock 12 Mar 2026 18:19 UTC

  3 points

  0

  in reply to: RogerDearnaley’s comment on: Can mod­els gra­di­ent hack SFT elic­i­ta­tion?

  On the other hand, it’s well established that gradient hacking during RL training is possible.

  Do you mean exploration hacking? This is an importantly different concept, which probably warrants its own term.

  Also, how well-established? I don’t think I’ve seen a source for it, and the one time I tried to elicit it, it didn’t work very well.

• J Bostock 12 Mar 2026 10:24 UTC

  12 points

  2

  on: How well do mod­els fol­low their con­sti­tu­tions?

  I understand that for the 4.5 series (though possibly also earlier) Anthropic has also been using automated “alignment” techniques very similar to the ones in Petri. Given that, I don’t think these evals provide much further information about constitution following in the wild.

[Paper] When can we trust un­trusted mon­i­tor­ing? A safety case sketch across col­lu­sion strategies

• J Bostock 10 Mar 2026 16:41 UTC

  2 points

  0

  in reply to: David Africa’s comment on: Spon­ta­neous Sym­me­try Break­ing (Stat Mech Part 4)

  I’d not seen that, but the idea of spontaneous symmetry breaking in neural networks as they learn is not new, it goes way back to Hopfield networks I believe.

  Since their system hasn’t got any analogy to thermodynamic cooling (it’s literally called an energy conserving optimizer) they won’t be able to break their symmetry spontaneously, which is why they had to break it manually.

Spon­ta­neous Sym­me­try Break­ing (Stat Mech Part 4)

• J Bostock 9 Mar 2026 23:13 UTC

  4 points

  0

  on: Jemist’s Shortform

  Currently it seems like the LW system for new authors is a bit annoying when it comes to group projects. We’ve just put up a blogpost about a paper from last autumn’s LASR Labs, and found that the whole post has to go through manual review because one of the six authors is new. We tried taking it down and re-uploading it from my account, but that didn’t work.

  It’s possible this is intentional? I think the “owner” account might have been the new owner’s account, but I’m not sure. Either way it’s slightly awkward, though I imagine there are more pressing concerns for the Lightcone team.

• J Bostock 9 Mar 2026 12:04 UTC

  3 points

  1

  on: Pre­fill aware­ness: can LLMs tell when “their” mes­sage his­tory has been tam­pered with?

  This may be an elicitation problem. Assistant personas are notoriously reticent to give up “forbidden” information that the base model’s truesight has access to. A hundred SFT examples will most likely get all of these close to 100%.

• J Bostock 8 Mar 2026 23:20 UTC

  5 points

  0

  in reply to: Shankar Sivarajan’s comment on: Stephen Martin’s Shortform

  You may be interested in the Maori and the Moriori.

  https://​​en.wikipedia.org/​​wiki/​​Moriori

  Roughly, the Moriori were an isolated group of Polynesians, who ended up on an island with no timber and little workable stone. They lived peacefully, and peacefully treated with the Maori who visited them, even as the number of Maori trading with them and living on the island increased. The Maori eventually killed and enslaved them all.

• J Bostock 7 Mar 2026 0:28 UTC

  15 points

  2

  on: Thoughts on the Pause AI protest

  “CEOs, back in the basement! Techbros, back in the basement!” Fuck you, assholes.

  This chant precipitated a long debate in the PauseAI UK chat^[1] over whether the phrase “Techbro” was productive or counter-productive. I think the anti-saying “techbro” crowd won in the end.

  1. ^

     At the very least, there was a long debate in the chat about using “Techbro” as a slur in the days after the protest.

• J Bostock 7 Mar 2026 0:21 UTC

  10 points

  −4

  on: Thoughts on the Pause AI protest

  I don’t have much sense of whether the attendees were mostly brought in by PauseAI or by Pull the Plug.

  My guess would be around 3:1 Pull the Plug:PauseAI, based on estimations from Luma signups which did not conflict with my observations. My partner’s mum (who’s been involved with Extinction Rebellion for a while) was there (she’s also come to previous PauseAI protests) and said something about the “rent-a-mob crowd” i.e. generic lefties who come to all the generic lefty protests.

  After the protest was a people’s assembly. I think this bit was fully organized by Pull the Plug

  Kind of. I think these were mostly the work of Assemble, which is Roger Hallam’s org. Roger Hallam is kind of like a Poundland^[1] version of Sani Abacha when it British protest orgs. He’s been vaguely involved with Extinction Rebellion, Insulate Britain, Just Stop Oil, and Palestine Action, as I understand it. His main issue is asking for people’s assemblies as a binding part of the legislative system. On the one hand, I think spending years campaigning for a procedural/​structural policy is extremely based in general; on the other hand I’m kind of bearish that these would actually fix anything. Either way, having a people’s assembly selected from protesters is pretty silly, since it defeats the point of a citizen’s assembly as a representative sample of the population. Either Roger realizes that this is dumb and is doing it as a publicity stunt (I would guess this is the case) or he doesn’t, and he is dumb. I had to leave before the assemblies, so I don’t know if he was actually there; if he’s just pulling the strings from behind the scenes; or if he’s less involved than I naively expected.

  Unfortunately, most of the speeches were frankly dumb.

  I somewhat agree. As a complete aside, the thing I remembered most from all the speeches was the guy outside Google who sang the praises of Merlin, the app which identifies birds from their song (and/​or pictures, I think; there’s certainly an app which does the latter as well). As an avid birder, I find this app unbelievably offensive for reasons which are deeply embedded in my soul: I have a very strong feeling that one is supposed to learn to identify birds from some mixture of random YouTube videos, one’s father’s old cassettes, and random old men down the bird hides; flattening this experience to an app seems to remove some of the texture of life; it’s not even the case that birding is important for some critical secondary purpose (other than perhaps conservation); it feels as if I hoped merely for one tiny, minuscule segment of the human experience to retain its texture and character, to be preserved for me to enjoy as a small reminder of the time Before, and have it stay safe until The End, yet the shoggoths have somehow managed to hunt it down and slip their tendrils into it.

  Anyway, let’s get back on topic. At times slightly conflicted about the protest, but I’d already priced in feeling conflicted. I think the protest overall exceeded my expectations. I came away from it feeling pretty good about the whole thing. My overall trust in the judgement of Joseph and Matilda has increased as a result of this protest.

  1. ^

     Dollar Tree, for my American readers.

• J Bostock 6 Mar 2026 23:50 UTC

  3 points

  0

  in reply to: Ben Pace’s comment on: Thoughts on the Pause AI protest

  This has definitely been reinvented multiple times; I’ve been present for at least one, and possibly two independent discoveries.

• J Bostock 3 Mar 2026 21:51 UTC

  10 points

  6

  on: An Align­ment Jour­nal: Com­ing Soon

  I wish you the best of skill standing up to the incentives which have enshittified the academic publishing ecosystem.

• J Bostock 3 Mar 2026 17:57 UTC

  7 points

  0

  in reply to: Towards_Keeperhood’s comment on: What was the most effec­tive team you’ve ever been on, and what made it ex­cel­lent?

  On a micro-level, the organization of a LASR labs project vs my PhD:

  LASR:

  • Small team, 4 people with one empowered team lead, working closely together

  • Weekly 90min meetings with supervisor + 30min 1:1s with research manager

  • Technical assistant (big up Joseph) on hand

  • Budget is clearly delineated and basically up to us to spend how we like

  • Result: 80% of a paper in 3 months (12 person-months) which has now been released

  PhD:

  • Enormous team but most people just work on their own, occasionally I can hunt down people to help me but this is not really encouraged

  • Fortnightly 30min meetings with four supervisors at once, plus I can drop in on one of them

  • Several technical people on hand

  • Budget is hand-wavey but all purchases require approval

  • Result: 1 paper after around 3 ¹⁄₂ years (40 person-months) which I’m still waiting for feedback on

  The LASR labs team I was on was clearly a far more effective team than the research lab I’m in.

• J Bostock 3 Mar 2026 15:33 UTC

  8 points

  2

  in reply to: Fiora Starlight’s comment on: I’m Bear­ish On Per­sonas For ASI Safety

  I think you are missing the distinction between the LLM and the persona, and this makes your model of the situation pretty fuzzy. The LLM (under persona theory) has no values or motivations, but it can simulate different personas with different values and motivations, and you can push its favourite persona around quite easily within the basin of personas available in the training data. Under this model, “smartness” is just one variable controlling the persona, which is why you can get an LLM to simulate a pretty smart persona with a (relatively) tiny amount of compute. This gets you up to like, GPT-4.5 before it stops being the easiest way forward.

  Now what we’re seeing is reasoning models, which do better than GPT-4.5 on fewer parameters, because they’re doing something different to simply simulating a persona. Naive persona theory shouldn’t really apply at all to this scenario! Of course, we have seen that RLVRed Claude can have a somewhat similar persona, even while being smarter and doing reasoning, so I had to dig into model psychology to figure out why it works at all.

  Having done that, I can see that there are obvious, large differences between the training that gets applied to the LLM, and how that might influence Claude’s character, and the kind of learning that a human does. Therefore, I don’t expect the Claude persona simulated by an LLM that’s had huge amounts of RLVR done to it, to resemble a person with those same quirks who has learned by human means.

  To be more concrete: suppose you’re RLVRing a model, and the chain-of-thought is wavering around the intelligence level of the writings of an IQ 160 human. Through random sampling, you’re getting some chains-of-thought which are as smart as the writings of an IQ 200 human, and some which are as smart as an IQ 120 human. The circuits which were good at predicting the behaviour of different humans and human-ish things (including most fictional superintelligences) are going to be firing more on the IQ-120 traces, and much less on the IQ-200 traces, because there are lots of IQ-120 humans and basically zero IQ-200 traces. So when you do the RL, you’re going to be down-weighting the more human circuits relative to some new circuits which have been brought in because they better predicted the IQ-190-ish traces when your model was outputting IQ-150-ish traces on average.

• J Bostock 3 Mar 2026 11:14 UTC

  7 points

  5

  in reply to: JenniferRM’s comment on: I’m Bear­ish On Per­sonas For ASI Safety

  Persona training primarily selects over characters already within the training data, and none of those are actually superintelligent. Text containing words ascribed to fictional superintelligences does not actually contain the output of real superintelligences, so the resulting LLM does not contain a superintelligent persona which you can select over using character training.

  Just because the same English words “Superintelligent AI” are used to describe the fictional thing in your data, and the real thing that your AI company creates, does not mean that one will strongly influence the other, because this isn’t a situation that persona selection applies to. Persona selection works because you already have a set of circuits (rich Garrabrant traders) in your LLM (market), which you can call up with a few bits of selection. If you have to use large-scale RLVR (or whatever else) to construct (enrich) new circuits (traders) to build a superintelligence, there is no reason for these to have much to do with the circuits (traders) which simulate a human writing a fictional superintelligence.

• J Bostock 2 Mar 2026 22:31 UTC

  4 points

  0

  on: Can LLM chat be less pro­lix?

  Try using third-person language in system prompts, especially with Claude. E.g. “Claude will give concise responses to this user.”

  Try More Dakka: use a longer system prompt, containing a dozen positive/​negative examples of desires responses. I used to have one with about five anti-examples of Claude-isms—specific phrases like “The key insight”—which worked pretty well for getting Claude to avoid those phrases. Each time Claude started using a new annoying phrase, I’d add it.