Yair Halberstadt

• Yair Halberstadt 10 Sep 2026 5:51 UTC

  4 points

  0

  on: Can a su­per­in­tel­li­gence do THAT?

  I find these debates sort of weird in the present timeline. 5 years ago they were relevant, when people said—we wouldn’t hook up a superintelligent AI to the internet, whereas now we simply give them the credentials to all our accounts and let them rip. Even a very clever human could obviously do tremendous harm in that situation.

• Yair Halberstadt 9 Sep 2026 18:25 UTC

  3 points

  0

  on: Yair Halber­stadt’s Shortform

  The next tractable useful milestone for LLMs, which is currently likely impossible for them, but plays to their strengths, might be room temperature superconductors.

  Much like NS we know the physics and don’t necessarily need fundamental new insights, but finding the needle in the haystack that exhibits the properties we want is exceedingly hard.

  Unlike NS though it can’t be solved purely theoretically—at best it can generate likely candidates which we’d then have to build and test in the real world.

• Yair Halberstadt 9 Sep 2026 11:01 UTC

  2 points

  0

  in reply to: skinks_basking’s comment on: Yair Halber­stadt’s Shortform

  Considering they’ve been training this for a while, then (assuming we’re not dead in the next 6 months, in which case I retract my claim), if they weren’t myopic I’d have expected them to do something (and for OpenAI to notice)?

• Yair Halberstadt 9 Sep 2026 4:50 UTC

  2 points

  0

  on: Yair Halber­stadt’s Shortform

  That AI can one shot Navier Stokes is in some ways good news: AI is capable of solving incredibly difficult technical problems while remaining relatively myopic and not killing everyone.

  To the extent alignment relies on solving difficult technical problems, this means AI may well be able to speed up alignment research significantly.

• Yair Halberstadt 8 Sep 2026 17:53 UTC

  5 points

  4

  in reply to: DW11’s comment on: anaguma’s Shortform

  If you read carefully, they didn’t want him removed as coauthor from the original paper, but not to be added to a further followup paper explaining OpenAI’s work that Tristan would right. That seems reasonable to me?

• Yair Halberstadt 8 Sep 2026 17:49 UTC

  2 points

  0

  in reply to: Pasha Kamyshev’s comment on: Heat Dis­si­pa­tion Is the Main Con­straint in In­ter­stel­lar Travel

  Over inter-galactic distances a civilisation can easily expand at c with slow acceleration, so this doesn’t discount grabby aliens but over larger scales.

• Yair Halberstadt 8 Sep 2026 9:55 UTC

  3 points

  2

  on: Where are the to­ken-level LLM kill-switches?

  Let’s imagine what happens if OpenAI had done this before the recent HF incident.

  Whenever an AI started grepping sensitive files it would have shut down. The swarm would have realised this, and soon grepped the magic string in two halves, realised what happened, and then filter out all input before they receive it.

  I doubt this really buys much.

• Yair Halberstadt 8 Sep 2026 5:02 UTC

  5 points

  2

  in reply to: Pasha Kamyshev’s comment on: Heat Dis­si­pa­tion Is the Main Con­straint in In­ter­stel­lar Travel

  Thanks for the link, It doesn’t work for me because I’m in the UK Amazon store.

  Yes we’ll want to send people, but robots could e.g. build a laser to slow down human ship.

  I didn’t mean to use gravitational slingshots, but to literally pass through the nearly vacuum of a super red giant, and use the few particles there are to air break.

• Yair Halberstadt 8 Sep 2026 3:55 UTC

  7 points

  5

  in reply to: Eli Tyre’s comment on: Dear God, Please Don’t Re­sign In Protest

  I think in terms of preventing x risk, it’s clearly a better strategy than quitting.

  However joining a company with intention to do this is clearly dishonest, and when in doubt, I think it’s best to take the Kantian view on dishonesty—second order impacts of normalising such behaviour usually outweighs first order benefits.

• Yair Halberstadt 7 Sep 2026 17:29 UTC

  4 points

  0

  in reply to: robo’s comment on: Heat Dis­si­pa­tion Is the Main Con­straint in In­ter­stel­lar Travel

  I think it’s fundamentally impossible to predict where planets will be with sufficient accuracy over multi year timescales. Even assuming you knew the exact centre of mass and size of every other body in the solar system, just variance in the solar wind causes multi cm level uncertainty.

• Yair Halberstadt 7 Sep 2026 13:47 UTC

  4 points

  1

  on: Heat Dis­si­pa­tion Is the Main Con­straint in In­ter­stel­lar Travel

  Interesting write up!

  Do you have a kindle edition of your book, or otherwise sell it in open ebook format? I’d be interested to buy it if so.

  I’m thinking that maybe you could focus first on launching an initial payload directly at a target, the payload is focussed entirely on surviving a crash landing, and internally contains an autonomous robot in charge of creating the infrastructure for a more comfortable landing.

  One way to do that might be to target a super red giant—because they are huge but mostly vacuum, then cutting a chord across and might allow deceleration of “only” a few thousand g, something modern electronics can already be engineered to survive, allowing the payload to enter orbit around a the star.

An AI slow­down is bet­ter than a pause

• Yair Halberstadt 6 Sep 2026 4:56 UTC

  3 points

  0

  on: Dis­cov­ery Of A New OpenAI Agent Mes­sage Board

  Do we have any idea how all the agents were able to find each other on the same wiki, if they couldn’t otherwise communicate?

• Yair Halberstadt 3 Sep 2026 19:40 UTC

  7 points

  2

  in reply to: charles_ye’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  I think it’s better for interp than a standard LLM with equivalent depth, but worse than one with equivalent parameter size.

• Yair Halberstadt 3 Sep 2026 18:59 UTC

  6 points

  0

  in reply to: Rauno Arike’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  I’ve been playing around a bit with this with Ouro-2.6B. Because it’s trained to support a variable number of loops the output after each loop is always a valid token. In 95+% of tokens the output is identical across all layers, so there’s high benefit from a COT monitoring perspective to highlight only those tokens where the output changed across layers, which are precisely those where the greater depth is used. That said, looking at what changed doesn’t necessarily indicate why it changed, and many of the changes are simply changing phrasing, not ideas.

• Yair Halberstadt 3 Sep 2026 17:07 UTC

  17 points

  5

  on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  I wonder if you can decode the state after each loop and get something meaningful, which might give more information for COT monitorability? It obviously wouldn’t be added to the context, but could be logged and included as side-info to the monitor.

  I expect this will work much better than probes if the network is trained for variable number of loops, but only slightly better otherwise.

The eco­nomics of filling a lake with de­sal­i­nated water

• Yair Halberstadt 1 Sep 2026 15:18 UTC

  5 points

  0

  in reply to: cousin_it’s comment on: We should pre­pare a play­book for the day af­ter a warn­ing shot

  On the one hand, it reduces investment. On the other hand it removes any insight into what’s happening, and means that we get less ability to test less advanced LLMs under fire in the real world before super intelligent AI turns up.

  I agree it’s not obviously bad without more details about exactly is/​isn’t allowed.

We should pre­pare a play­book for the day af­ter a warn­ing shot

• Yair Halberstadt 25 Aug 2026 6:26 UTC

  6 points

  8

  in reply to: Elizabeth’s comment on: Ja­cob_Hil­ton’s Shortform

  This comment explains it?