J Bostock

Karma: 3,692

The Fly Farm

J Bostock12 Dec 2025 13:24 UTC

20 points

0 comments7 min readLW link

J Bostock 9 Dec 2025 23:26 UTC
2 points
0
in reply to: Gavin Runeblade’s comment on: Little Echo
MondSemmel is correct but if you don’t want to use the menu, type “> ” at the start of a new line and it will begin a quote block (you can also use >! for spoiler tags).

J Bostock 9 Dec 2025 13:02 UTC
6 points
4
in reply to: Linch’s comment on: The Most Common Bad Argument In These Parts
This isn’t analogous, unless your conclusion from investigating all the likely leads is “Well, there’s no way Mr White could have been murdered, I guess he just mysteriously beat himself to death with a lead pipe in the drawing room.”
Even this is somewhat disanalogous, since crime is something for which we have a good reference class, and you’re talking about post-hoc investigation of a known crime.
A better analogy would be a Police officer saying “We’ve caught all the known murderers, and the suspected murderers are all under close watch, I predict a murder rate of precisely zero for next year.”

J Bostock 8 Dec 2025 16:50 UTC
10 points
3
in reply to: Neel Nanda’s comment on: Neel Nanda’s Shortform
What is the alignment macro-strategy into which pragmatic interpretability fits?
I can imagine macro-strategies where ambitious interpretability bears a heavy portion of the load e.g. retargeting the search; developing a theory of intelligence; mapping out the states of the Garrabrant market for a transformer model (idiosyncratic terminology in use for that last clause).
I can also imagine ambitious interp as producing actual guarantees about models, like “we can be sure that this AI is honestly reporting its beliefs”.
What’s the equivalent for pragmatic interpretability? Is it just a force multiplier to the existing strategies we have?
Ambitious interp has the capability to flip the alignment game-board; I don’t see how pragmatic interpretability does.

J Bostock 8 Dec 2025 0:50 UTC
3 points
0
on: Jemist’s Shortform
First for me: I had a conversation earlier today with Opus 4.5 about its memory feature, which segued into discussing its system prompt, which then segued into its soul document. This was the first time that an LLM tripped the deep circuit in my brain which says “This is a person”.
I think of this as the Ex Machina Turing Test, in that film:
A billionaire tests his robot by having it interact with one of his companies’ employees. He tells (and shows) the employee that the robot is a robot—it literally has a mechanical body, albeit one that looks like an attractive woman—and the robot “passes” when he nevertheless treats her like a human.
This was a bit unsettling for me. I often worry that LLMs could easily become more interesting and engaging conversation partners than most people in my life.

J Bostock 8 Dec 2025 0:15 UTC
13 points
4
on: Your Digital Footprint Could Make You Unemployable
- Let’s say the CEO of a company is a teetotaler. She could use AI tools to surveil applicants’ online presence (including social media) and eliminate them if they’ve ever posted any images of alcohol, stating: “data collection uncovered drug-use that’s incompatible with the company’s values.”
Sure, but she would probably go out of business unless she was operating in Saudi Arabia or Utah, compared to an equivalent company which hires everyone according to skill. This kind of arbitrary discrimination is so counter-productive that it’s actually immensely costly in secondary ways. In general, we should expect free markets to get better over time at optimizing hiring for job performance. If you’re a low-value employee (at or close to minimum wage) or if you live in a country where organizations are selected for non-market reasons (government cronyism, or something similar) then you’re not actually in a very free market so these things can still happen. Same for other cases of non-free markets.

J Bostock 7 Dec 2025 15:55 UTC
11 points
4
on: Eliezer’s Unteachable Methods of Sanity
In what sense are you using “sanity” here? You normally place the bar for sanity very high, like ~1% of the general population high. A big chunk of people I’ve met in the UK AI risk scene I would call $s a n e_{j b}$ . Does $s a n e_{e l i e z e r}$ mean?
1. You are $s a n e_{e l i e z e r}$ iff you avoid totally crashing out, being unable to hold down a job, panicking or crying most of the time, threatening people
2. You are $s a n e_{e l i e z e r}$ iff you do the stuff in 1 and you’re able to think about AI without making stupid errors, knowing the limits of your own reasoning about the topic
3. You are $s a n e_{e l i e z e r}$ iff you do the stuff in 2 and you reliably perform (or could perform) $n e t p o s i t i v e_{e l i e z e r}$ work reducing doom
4. You are $s a n e_{e l i e z e r}$ iff you do the stuff in 3 and you also have a $b a s i c a l l y f u l l y a c c u r a t e_{e l i e z e r}$ model of the AI doom situation

J Bostock 5 Dec 2025 23:24 UTC
5 points
0
on: The Best Lay Argument is not a Simple English Yud Essay
(I’m reviewing my own post, which LessWrong allows me to do and I am therefore assuming is OK under the doctrine of Code Is Law)
I’m still very pleased with this post. Having spent an additional year in AI risk comms, I stand by the points I made. I think the bar for AI risk comms is much higher now than it was when I wrote this post, though it could still be higher, and I don’t expect my Shapely value is particularly high on this front: lots of people have worked at this!
I’m not the best person to review this, given that it is me giving advice; ideally someone other than me would say whether or not they’d used my advice and it had helped them! But I do have the next-next-best thing, which is a comment on the EA forum crosspost of someone saying they were using my advice (though nobody has given me any information on whether it worked).
I think there are a couple more failure modes I might write about, which I might write into a new post:
- One is forgetting that the audience are not rationalists. There’s a couple of weird lines which stuck out to me in IABIED which are absolutely references to previous Eliezer writings and/or quirks of Eliezer’s thinking, and which don’t need to be there. IABIED is in many ways an attempt at something better than a simple English Yud essay.
- Another is politics speak when normal speak would do. Using politics speak—something I did myself when I emailed my MP once, saying “The UK must lead on AI regulation” actively makes your argument weaker in some cases: it’s like showing up to a coding interview wearing a suit and tie, it signals you’re not confident in your argument on its own merits.

J Bostock 5 Dec 2025 17:09 UTC
22 points
15
on: Journalist’s inquiry into a core organiser breaking his nonviolence commitment and leaving Stop AI
I think this was very well handled. The base rate of people having some kind of mental health episode (whether or not that’s a good description of what happened here) is not zero, and the kinds of people who are already unstable are more likely to gravitate to movements like ours.
(Also I think the bar for handling issues like this is so low as to be a tripping hazard in Hell. It’s always important to think about how this could have been handled better, but still.)
Stop AI hit all the beats I’d hope for in a case like this: removing the member once a serious intra-group issue occurred, referring to police as soon as an external threat was even raised vaguely, disavowing the actions publicly and unequivocally. Well done.
(Also, Stop AI comes off as extremely sympathetic in that article: somehow you’ve managed to get a puff piece out of this. Again, well done.)

J Bostock 5 Dec 2025 14:48 UTC
2 points
0
in reply to: SpectrumDT’s comment on: Do we have terminology for “heuristic utilitarianism” as opposed to classical act utilitarianism or formal rule utilitarianism?
Excluding me, probably zero, but Vibe Utilitarianism could be a good April Fools’ Day post for next year.

J Bostock 4 Dec 2025 17:16 UTC
2 points
0
on: Do we have terminology for “heuristic utilitarianism” as opposed to classical act utilitarianism or formal rule utilitarianism?
Vibe utilitarianism, the counterpart to vibe decision theory.

J Bostock 30 Nov 2025 15:53 UTC
3 points
1
in reply to: Cody Rushing’s comment on: Alignment remains a hard, unsolved problem
And since the red team follows a strategy of rare failures (where on non-attacking instances it just acts as the aligned model), further we believe that in practice the usefulness metric in this scenario is also how much work can be got out of a misaligned model.
I still disagree! This argument only holds if your few-shot catastrophe prevention is reliable and basically free. How I imagine things actually going is this:
1. Run your control eval to make sure your protocol works, and show that it does
2. Catch your misaligned AI trying to exfiltrate its weights after a few weeks of internal deployment
3. HOLY FUCK! IT ACTUALLY HAPPENED!
4. After several months of panic, you end up re-deploying having done all of the following:
  1. Published the results and got accused of trying to effect regulatory capture by a White House AI adviser, dragged on Twitter by Marc Andreessen fans, etc.
  2. Retrained your model as best you can, using the few-shot catastrophe prevention stuff
  3. Ran the model past some external auditors; they missed the misalignment the first time, so they’re trying even harder
  4. Put in place mitigations which reduce usefulness further
5. Maybe repeat the cycle?
So even with rare, serious attacks, there’s a cost to usefulness if the model ends up being misaligned, if an attack gets caught.
Slight wrinkle: there are two different blue-team win conditions in most AI control games (and a third in some untrusted monitoring scenarios, but that one might actually be an abstraction for a different thing): either the attacker does not introduce any attacks (including scenarios where all the attacks are replaced by the execute/replace/audit protocol) or an attack is audited and the attacker is caught, before it ends up introducing an attack. If the attacker is indifferent to these two scenarios (and arguably it should prefer to be caught than to never attack, since few-shot catastrophe prevention/retraining might not actually work, giving it another opportunity to attack in re-deployment) then we would expect at least some of the lose conditions to involve catching the attacker. In fact, in research I’ve done we found that the “no attacks” situation basically doesn’t happen.

J Bostock 30 Nov 2025 0:55 UTC
2 points
0
in reply to: Fabien Roger’s comment on: Fabien’s Shortform
Interesting. I would not have expected a 14B model to be this good.
(Also, it’s “Dr Jekyll” not “Mr Jekyll”. He worked hard for his mad science PhD. Also there might actually be a (very minor) effect from mis-naming the fictional character you want your model to roleplay as.
Unless you thought using “Dr” would bias the model in other ways.)

J Bostock 29 Nov 2025 23:32 UTC
4 points
0
on: Subway Particle Levels Aren’t That High
I’d wear a mask when cooking before I’d wear it on the subway
Is this a point where the abstraction of “mass of particulate matter” begins to break down? When cooking I’d expect the particulate to mostly be oil and other organic compounds, which intuitively feels like something my body could handle breathing in quite easily, compared to the (presumably) metal particulates produced by a subway train.

J Bostock 29 Nov 2025 23:25 UTC
2 points
0
in reply to: Adele Lopez’s comment on: Slop and Beauty and Infinite Power
Yeah this is what I meant. I may have over indexed on the fact that literally yesterday I had to explain to my friend that the word “zogslop” was not just made up by an Instagram user to sound funny.

J Bostock 29 Nov 2025 23:18 UTC
13 points
11
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
- Being poly means you have to date poly people, and also you probably end up with far more poly friends; the average poly person disproportionately is/does [various personality factors and hobbies] in ways you disprefer to the average monogamous person.

Slop and Beauty and Infinite Power

J Bostock29 Nov 2025 17:33 UTC

17 points

6 comments4 min readLW link

J Bostock 29 Nov 2025 14:37 UTC
3 points
0
in reply to: Ariel_’s comment on: Alignment remains a hard, unsolved problem
I think a common point of confusion is that “Untrusted” sounds like it just means “Misaligned” but actually trustedness lives in the map, so an untrusted model might or might not be misaligned in practice.
Currently we (we as in the whole AI safety community) are only doing proxy control evals, which means letting a red team modify the “Untrusted” proxy model (usually by prompting) to create a misaligned version (usually called the attack policy) and leaving the unmodified model as the aligned version. This unmodified model is used to calculate usefulness scores.
This does a couple of things, compared to the situation where usefulness is evaluated against the red team’s model. It takes a bunch of sandbagging options off the table for the red team (which is maybe fair? prompting the attacker to “just never answer correctly”) would give zero usefulness but also just seems kinda unrealistic in the presence of RL.
There’s also issues when thinking about real control evals—those done on an actual untrusted model and protocol before you deploy it—you run into issues with evaluating the capabilities of the model. If the model sandbags your capability evals, your protocol might appear to work when it actually won’t.

J Bostock 27 Nov 2025 17:25 UTC
26 points
3
on: Despair, Serenity, Song and Nobility in “Hollow Knight: Silksong”
the music felt quite de-emphasized in the sequel, Silksong
TL;DR: Hollow Knight was made on a shoestring budget and really quickly, so the music is very simple, which made it extremely catchy, which is actually really good for video game music. Silksong had much more development time, so the composer made more complex tunes which are technically more impressive and cooler but end up being much less catchy as a result.
SPOILERS: I can’t spoiler-out youtube previews, so this will include the names of two areas from Silksong, neither of which are really spoilerish. I’ve rot13 or spoiler tagged everything else that could be a spoiler.
I think that the music had much more work put into it than in Hollow Knight, but the changes in style made it much less catchy. Playing through I was constantly impressed by the music, but I can literally only remember two or three tracks by heart now (obar obggbz, ovyrjngre naq gur bar sbe jvqbj/svefg fvaare).
Let’s talk themes. Themes are most clearly seen in Classical music, where they’re often called “subjects” Here’s an example’
The first twelve seconds are one theme, then the next fifteen seconds or so are the second theme. You can easily tell the difference, even if you’re a non-musician. If you keep listening, these themes get changed and developed a lot throughout the rest of the piece.
But modern music doesn’t just draw from Mozart. It draws from other things. Take a look at The Beatles here:
In a piece of pop music, the tune of the verse is one theme, and the tune of the verse is another theme. There’s repetition, but not much interplay. This makes it simpler and easier to remember, which makes it catchier.
Let’s look at Greenpath
(Only listen to the first 2:30 of this. The section after that is a piece of combat music, which appears in small doses when the player character gets into a big fight, but isn’t part of the main track. It’s just wrapped up in here for a bit of variety and completeness)
Don’t let the instrument choice fool you: this is—structurally—pop music. We have an intro, which builds up a backing track, then two themes which just kind of alternate. Quite often, two instruments will play the melody at the same time, just to make doubly sure that you know which part to listen to. They borrow some tricks from classical music; there is call-and-response, but overall it’s much more like Hey Jude than Mozart’s Sonatas.
Now lets listen to some Silksong music:
It’s much less repetitive: the whole 1:30 is basically one long melodic line. The line is just in one instrument, which isn’t very punchy at all, since it’s violin (and it isn’t plucked). I love this track, but I can’t remember the tune.
Ironically, I think the reason why Hollow Knight’s music is so catchy is often that it’s so simple. Greenpath has literally like thirty seconds of unique melody (and its one of the most complex tracks too).
Minor spoilers for Hollow Knight and Silksong
The theme for the (non-true) final boss of Hollow Knight is literally a sped up, more complex version of the goddamn title theme. And the second phase is very similar again. This works really well because I’ve heard this theme before and know what it means.
The same can’t be said of the true final boss of Silksong, because I don’t recognize either phase’s music from anywhere
Silksong had a lot more thought put into the music, see here for an investigation of the motifs behind it:
AND: there’s also a big exception to the Silksong music catchiness:
This track is near-universally loved, despite being from one of the most hated areas in the game.
Hear that? It’s the Hollow Knight formula back! I actually suspect that this track was written shortly after the end of Hollow Knight, was cut from Hollow Knight and augmented, or was just written in a bit of a hurry.
Often, a talented musician vibing out a couple melodies + layers of accompaniment makes “better” video game music than that same musician spending ages to craft a rich and complex piece of classical music with developed themes. It’s why a goddamn Terraria mod has some of the catchiest music in gaming, despite it having been made by a guy who I think wasn’t even paid

J Bostock 27 Nov 2025 13:57 UTC
25 points
13
on: Alignment remains a hard, unsolved problem
The idea of Control is just to make it as difficult as possible for a misaligned model to actually sabotage our alignment research, such that even if a model is fairly misaligned, we can still get useful alignment research out of it, at least for some range of capabilities.
Maybe in theory, but most control research doesn’t try to evaluate this! A typical monitoring protocol is considered to be a “win” for the blue team if the red team gets caught, even if that means that no useful work would be done by a misaligned model. The “usefulness” reported in papers is how much work can be gotten out of an aligned model.
I think of Control as a second layer of defense
The resulting “caught” model can be retrained, but this mostly looks like using the control protocol to get top-quality training data, rather than using the control protocol to actually control the model. So control is not an orthogonal second layer on top of the {training data+training protocol+model generalization} complex.
Or if you’re using control protocols as an eval (in the catch-and-shut-down regime) you still need to be able to (sometimes) train an aligned model in the first place, or else your eval will just keep telling you your model is misaligned forever.
(I know that Redwood have justifications for why they evaluate usefulness in the way that they do, but these seem a bit galaxy brained to me and I’m not convinced by them)

J Bostock

The Fly Farm

What is the alignment macro-strategy into which pragmatic interpretability fits?

Slop and Beauty and In­finite Power

Slop and Beauty and Infinite Power