J Bostock

• J Bostock 12 Oct 2025 20:58 UTC

  3 points

  0

  in reply to: Fabien Roger’s comment on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  I lean towards all epistemic environments being adversarial unless proven otherwise based on strong outside-view evidence (e.g. your colleagues at a trading firm, who you regularly see trading successfully using strategies they freely discuss with you). Maybe I’m being too paranoid, but I think that the guf in the back of your mind is filled with memetic tigers, and sometimes those sneak out and pounce into your brain. Occasionally, they turn out to be excellent at hunting down your friends and colleagues as well.

  An adversarial epistemic environment functions similarly to a normal adversarial environment, but in reverse. Instead of any crack in your code (therefore, a crack in the argument that your code is secure) being exploitable, the argument comes into your head already pre-exploited for maximum memetic power. And using an EFA is one way to create a false argument that’s highly persuasive.

  I also think that—in the case where the EFA turns out to be correct—it’s not too hard to come up with supporting evidence. Either a (good) reference-class argument (though beware any choice of reference class!) or some argument as to why your search really is exhaustive.

• J Bostock 12 Oct 2025 20:43 UTC

  5 points

  0

  on: Jemist’s Shortform

  I have no evidence for this but I have a vibe that if you build a proper mathematical model of agency/​co-agency, then prediction and steering will end up being dual to one another.

  My intuition why:

  A strong agent can easily steer a lot of different co-agents; those different co-agents will be steered towards the same goals of the agent.

  A strong co-agent is easily predictable by a lot of different agents; those different agents will all converge on a common map of the co-agent.

  Also, category theory tells us that there is normally only one kind of thing, but sometimes there are two things. One example sums and products of sets, which are co- to each other (the sum can actually be called the coproduct) there is no other operation on sets which is as natural as sums and products.

• J Bostock 12 Oct 2025 17:40 UTC

  4 points

  0

  in reply to: rotatingpaguro’s comment on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  Uhhh either I forgot the word and it means nothing, or it’s (roughly) the place where souls live before they get born in Judaism.

• J Bostock 11 Oct 2025 21:07 UTC

  14 points

  4

  in reply to: Ben’s comment on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  Interesting! I have a post cooking somewhere in the guf of my brain which is something like “How Much Imagination Should We Use?”. A “Proof by Lack of Imagination” is dual to a “Proof by Excess of Imagination”.

  Example: suppose someone says “I can imagine an atomic copy of ourselves which isn’t conscious, therefore consciousness is non-physical.” and I say “No, I can’t imagine that.”

  On the one hand, a good model of the world should prevent me from imagining things which aren’t possible. If I have a solid idea of what acidic and alkaline solutions are, I should have great difficulty imagining a solution which is both at once! But on the other hand, it is a good thing if my model can imagine things which are possible, like a power-plant which harvests energy from a miniature sun.

  I’ve been thinking about things from a pedagogical angle a lot lately, which is why this post is phrased in the specific way it is. I would like to come to a better conclusion of how to move disagreements forward. If I say to someone “It seems quite easy to imagine an AI smart enough to take over the world using a medium-bandwidth internet connection” and someone says “No I can’t imagine that at all”, then it seems like we just reach an impasse immediately! How do we move forward?

• J Bostock 11 Oct 2025 20:59 UTC

  2 points

  0

  in reply to: abstractapplic’s comment on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  Fixed. Lol.

• J Bostock 11 Oct 2025 20:58 UTC

  5 points

  −1

  in reply to: Mateusz Bagiński’s comment on: The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

  If you don’t have a systematic way of iterating your ideas, your method of generating ideas is just free-association. So making an argument from “exhaustive free-association” means you’re arguing that your free association is exhaustive. Which it never is.

• J Bostock 11 Oct 2025 16:35 UTC

  12 points

  1

  on: Ex­per­i­ments With Son­net 4.5′s Fic­tion

  This story peters out towards the end in a way that I don’t think yours normally do. The prose is definitely a little tighter than some of your work, and flows better. The humor here is kind of “smoother” in that there are more common, less funny jokes, but the overall themes and payoff are much less surprising and interesting.

  Examples: The Origami Men had a genuinely interesting and surprising ending. The Company Man lampooned EAs and Rats and the rest of the SF circuit in a way that was very very funny at points. The shrimp section of that was much funnier than any of this.

The Most Com­mon Bad Ar­gu­ment In Th­ese Parts

Maybe Use BioLMs To Miti­gate Pre-ASI Biorisk?

• J Bostock 9 Oct 2025 16:27 UTC

  2 points

  0

  in reply to: Guive’s comment on: Jan_Kul­veit’s Shortform

  What someone spends time writing is an important signal. More generally, what someone is interested in—and spend time on—is one of the most powerful signals as to what they’ll do in future.

  “Real views” is a bit slippery of a concept. But I strongly predict that most of their outputs in future will look like it’s taken “AI will happen and we need to be there” as a primitive, and few will look like they’re asking themselves “Should AI happen?”. Because the latter question is just not a thought that interests them.

• J Bostock 7 Oct 2025 15:56 UTC

  13 points

  2

  in reply to: robo’s comment on: abramdem­ski’s Shortform

  I think this misses the point, since the problem is^[1] less “One guy got made psychotic by 4o.” and more “A guy who got some kind of AI-orientated psychosis was allowed to continue to make important decisions at an AI company, while still believing a bunch of insane stuff.”

  1. ^

     Conditional on the story being true

• J Bostock 6 Oct 2025 19:59 UTC

  5 points

  2

  on: What can Cana­di­ans do to help end the AI arms race?

  Recently I was talking to someone about Pause AI’s activities in the UK. They asked something like

  “Even if Kier Starmer (UK Prime Minister) decided to pause, what would that do?”

  And I responded:

  “We would have the resources of an entire nation state pointed towards our cause.”

  Canada (all this applies to the UK too) has a lot more resources to throw around than MIRI, Lighthaven, OpenPhil, and Pause/​Control/​Stop AI all put together. Canada is a founding NATO member. If Canada’s government, and military intelligence leaders, were “Doomer-pilled” on AI, they could do a lot more than we have done.

• J Bostock 6 Oct 2025 9:46 UTC

  7 points

  4

  in reply to: Wei Dai’s comment on: Cole Wyeth’s Shortform

  I recently, rather embarrassingly, made a post with a massive error which an LLM would have found immediately. I seriously misread a paper in a way that cut/​pasting the paper and the post into Claude and asking “any egregious misreadings” would have stopped me from making that post. This is far too useful for me to turn down, and this kind of due diligence is +EV for everyone.

• J Bostock 5 Oct 2025 23:23 UTC

  6 points

  2

  in reply to: anaguma’s comment on: porby’s Shortform

  The presence of induction heads would be a decent answer. If you show a model …[A][B]...[A] It will often complete it with [B]. One specific mechanism which can cause this is well-studied, but the general reason is that this pattern is really common in lots of different types of pretraining data.

• J Bostock 4 Oct 2025 12:33 UTC

  17 points

  4

  in reply to: Alex_Altair’s comment on: Alex_Al­tair’s Shortform

  Books are quite long. I think we should go with the shortest possible description of his life.

• J Bostock 3 Oct 2025 16:00 UTC

  5 points

  0

  on: IABIED and Memetic Engineering

  I saw someone (cannot find source) say that “Eliezer wrote 300% of the book, and then Nate wrote −200%”. I’ve not read much of Nate’s writing other than on LessWrong, so I don’t know if he’s solely responsible for the change in tone, or if Eliezer was writing more normie-ish to begin with, or what.

• J Bostock 3 Oct 2025 13:50 UTC

  8 points

  0

  on: Jemist’s Shortform

  Shrimp Interventions

  The hypothetical ammonia-reduction-in-shrimp-farm intervention has been touted as 1-2 OOMs more effective than shrimp stunning.

  I think this is probably an underestimate, because I think that the estimates of shrimp suffering during death are probably too high.

  (While I’m very critical of all of RP’s welfare range estimates, including shrimp, that’s not my point here. This argument doesn’t rely on any arguments about shrimp welfare ranges overall. I do compare humans and shrimp, but IIUC this sort of comparison is the thing you multiply by the welfare range estimate to get your utility value, if you’re into that)

  (If ammonia interventions are developed that are really 1-2 OOMs better than stunning, then even under my utility function it might be in the same ballpark as the campaign against cage-free eggs and other animal charities)

  Shrimp Freezing != Human Freezing

  Shrimp stunning, as an intervention, attempts to “knock out” shrimp before they’re killed by being placed on ice, which has been likened to “suffocating in a suitcase in the antarctic”.

  I think that’s a bad metaphor. Humans are endotherms and homeotherms, which means we maintain a constant internal body temperature which we generate internally. If it drops, a bunch of stress responses are triggered: shivering, discomfort, etc. which attempt to raise our temperature. Shrimp are poikilotherms, meaning they don’t regulate body temperature much at all. This means they don’t have the same stress responses to cold that we do.

  (I also doubt that, given they have no lungs and can basically never “not breathe” in their normal environment, they’d experience the same stress from asphyxiation that we do, but this is weaker)

  I would guess that being thrown onto ice effectively “stuns” the shrimp pretty quickly, as their metabolism—and therefore their synaptic activity—drops with their body temperature.

• J Bostock 2 Oct 2025 9:54 UTC

  10 points

  3

  in reply to: Shankar Sivarajan’s comment on: “Pes­simiza­tion” is Just Or­di­nary Failure

  That seems roughly right but I don’t think chaos/​revolution is necessary for this thesis.

  Imagine the world sitting in a deep basin. You’re trying to get it out. You can try a revolution: launch it up into the air and hold your breath waiting for it to land.

  Or you can try, sisyphus-style, to roll it out. You might get a bunch of allies, who you try and lead along a narrow and precarious ridgeline to your destination. Then, when you’re halfway to the top, some might notice that there’s a much easier path requiring much less of a push, and so the world gets pushed down off your intended route along the mountain pass and into a new crater.

  The latter seems to have been the case with environmentalism and AI Safety. They pushed the world fairly gradually, but it only takes a few individuals to push the world downwards into a new equilibrium, for the upward pushers to lose that fight.

  (There’s a Yudkowsky tweet somewhere which says something like this, which I’ve basically based the entire metaphor here on)

“Pes­simiza­tion” is Just Or­di­nary Failure

• J Bostock 30 Sep 2025 23:08 UTC

  3 points

  −1

  on: Claude Son­net 4.5: Sys­tem Card and Alignment

  I think we have evidence that this is not the primary thing going on, because there are parts of the evaluation where the model lacks context, could easily face an identical situation in deployment, and thus can’t know it is being evaluated.

  This in particular is not strong evidence, since the optimal strategy for a schemer is usually misbehave only very rarely, when it’s very confident that it isn’t in an evaluation. This is because the number of deployment runs will be >> the number of evaluations. If it’s unable to tell, then it should probably just not misbehave in that instance.