Hastings

Karma: 2,429

Hastings 5 Jul 2026 22:10 UTC
4 points
2
on: The case for the fleshman
I like this fleshman framing as it lacks the following issue:
Steelmanning makes a great deal of sense in terms of first order effects, but what I observe is that once a diverse group is in the habit of steelmanning, the group is exploitable where its more effective to vaguepost in such a way that all the factions steelman you into their camp, than to clearly spell out your belief. Incentive gradients against clear writing are like, pretty bad.
Fleshmanning picks the median instead of the extreme and it seems like factions can agree on median, or at least not predictably disagree like they do when picking best, where this predictable disagreement gets them pwned

Hastings 4 Jul 2026 11:59 UTC
7 points
3
on: Lydia Laurenson: “The Inside Story of Leverage Research”
The moods are so missing that it almost reads as a literary piece with an unreliable narrator.
We were really determined not to start a cult, which is why we repeatedly watched the YouTube video how to start a cult.
We were not an intellectual monoculture: for diversity we had a bunch of neo reactionaries. The allegations of us being a Peter Thiel cult are unfounded.
We didn’t have any psychotic breaks. None. But if we did, that would be perfectly normal for our type of groundbreaking organization, and an understood danger of the process. But we didn’t have any.

Hastings 1 Jul 2026 16:15 UTC
3 points
0
on: When should you know the point?
Some things I can’t nail down the point of but do:

Voting. Correctly working out the point of an individual voting requires a working decision theory but even though I don’t have that I’m happy to basically pascal’s wager it. I picked this as an interesting example as I would have no qualms asking someone to help me go vote if I needed a ride or something.

I’m writing a new programming language, even though I’m not sure what the point is. On base rates the world doesn’t need this. I haven’t strictly asked for help with this before I work out what the point is, but I have definitely “Hey look at this thing I’m making”ed my friends with it enough that they could ask me why. I picked this as an example in the hopes that someone would ask for details, after which I would “Hey look at this thing I’m making” them.

Reddit: this doesn’t have a point and I would ask for help to not do this if I thought anyone could help.

Hastings 29 Jun 2026 13:33 UTC
3 points
0
on: Hastings’s Shortform
Working through “A monad is a monoid in the category of endofunctors,” I was able to learn the definitions of monoid, category, and endofunctor pretty easily and have been blocked on “in” and “of” for significantly longer. (vague claim that this generalizes)

Hastings 18 Jun 2026 22:36 UTC
4 points
0
in reply to: Hastings’s comment on: DW11′s Shortform
Credentials: I have beaten ultrasound with the ML stick until it yielded a few times ( https://scholar.google.com/citations?hl=en&user=O1xhOlUAAAAJ ) it was terrible and I have lasting resentment toward an imaging modality, though great fondness for all my colaborators. Many failed projects that did not make the google acholar

Hastings 18 Jun 2026 22:21 UTC
3 points
0
in reply to: DW11’s comment on: DW11′s Shortform
ultrasound is awful to work with in traditional medical image processing technologies and pretty darn bad in 2015-2025 convolutional medical imaging ai technologies. Magnetic Resonance is so much better when its the right tool for the job that replacing it with ultrasound for cost reasons is a tarpit. There are cases where ultrasound is better but these center on leveraging an extremely talented operator who is manipulating the probe manually, which you lose in any of these bath based approaches. I’d be surprised if this goes anywhere.
What links here?
- Links #4: 2026/06 Part 2 by papetoast (1 Jul 2026 0:43 UTC; 8 points)

Hastings 17 Jun 2026 23:13 UTC
4 points
0
in reply to: niplav’s comment on: niplav’s Shortform
Worth noting that the only way to get a pull request accepted to stockfish is to beat stockfish at a different form of centaur chess (manually modify stockfish and then have your changed stockfish beat the original in a series of games) and this happens regularly.

Hastings 17 Jun 2026 16:17 UTC
4 points
0
in reply to: Hastings’s comment on: Hastings’s Shortform
I guess the main useful insight here is that .01% successful escape attempt during training sounds very aligned, and 100,000 successful escape attempts during training sounds very not aligned

Hastings 17 Jun 2026 15:53 UTC
5 points
−5
on: Hastings’s Shortform
Fermi estimate: Lets say each training episode for Claude Mythos cost a dollar, and Anthropic spent a billion dollars post-training Mythos. Now, in 0.01% of training episodes, Mythos broke out of the training environment entirely to get useful data from the public internet, which is a billion * .01% = 100,000 requests. I wonder how fast this last number is growing? Could an entity like the NSA or CCP, with taps in enough internet infrastructure, detect 100,000 weird requests if it looked hard enough? Potentially an avenue for monitoring and verifying pauses, analogous to detecting nuclear weapons tests by monitoring for radiation leaks. Spotting the containment breaches from a frontier lab’s training run, without their cooperation, would be an excellent safety project I think- not an easy project, but easier than half the stuff we are trying. It would look a lot like “ordinary” agentic AI traffic, but coming from weird IP addresses, and imho likely pretty internally homogenous- from my understanding algorithms like the one used by deepseek attempt the same task a large number of times in a batch with different seeds to get a signal, which would produce bursts of traffic to answer the same question, as a first spitball of a signal. Would be much easier with cooperation from at least one frontier lab since they know what the real leakage from their training runs looks like, but could monitor all of them.

Hastings 15 Jun 2026 10:58 UTC
4 points
0
in reply to: Thomas Kwa’s comment on: TurnTrout’s shortform feed
Thanks for replying, this is an important update for me.

Hastings 15 Jun 2026 10:52 UTC
2 points
0
in reply to: Nissa Seru’s comment on: TurnTrout’s shortform feed
I have a very similar setup, and have found similar convenience instead of inconvenience: specifically, agents can compile or configure tools without bonking each other or me. I also usually run agents on a server instead of my laptop, just so that they don’t stop what they’re doing if I close my computer to walk to a coffee shop, and so there is very little additional friction to virtualizing since its ssh and tunnels either way.

Hastings 14 Jun 2026 12:22 UTC
9 points
−3
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
Are alignment researchers seriously not even keeping the ai in a virtual machine? This feels like one of those Hastings Is Not Living in Berkeley and so is Frequently Surprised that Things Weren’t Jokes moments.

Hastings 13 Jun 2026 11:37 UTC
4 points
0
on: Sympathy for both sides of the egregious misalignment debate
If LLMs are alignable, the question isn’t whether LLMs can scale to ASI, its whether 1) LLMs are the equilibrium, compute optimal way to be intelligent or 2) we can coordinate to stay off the equilibrium in the time between discovering it and working out how to align it. 1) seems galactic-ally unlikely to me so LLM alignment is entirely reliant on 2) (so we would be well served to develop that capability)
There’s a bit of flexibility in that the compute optimal way to build intelligence could be llm-like enough for alignment to generalize, but this seems unlikely- for example, LLMs pre rlvr were way easier to make behave but it is unthinkable to coordinate on not doing rlvr. I expect more rlvr-like innovations.

Hastings 11 Jun 2026 14:07 UTC
3 points
0
in reply to: Hastings’s comment on: Hastings’s Shortform
If the average citizen from the subpopulation that has an active interest in not reading AI text is exposed to so much text generated by your specific AI model that they develop a rage response to “not only… but also...” then the correct response is not to train out the not only but also- it is to produce less text intended for sending to citizens with no interest in AI!

This smells like an early use case for satisficing. Writing a decorated github readme only when nicely asked to do so by the repo owner is a super chill low impact activity associated with a tiny financial reward and it is remarkable that the hyperscalars have managed to latch on to and Sorcerors-Apprentice-Water-Bucket this action and reward hard enough to enrage a real population of humans.

The response of trying to make the readmes less enraging instead of even pondering the option of making less of them does not bode well.

Hastings 11 Jun 2026 12:39 UTC
3 points
1
on: Hastings’s Shortform
An aligned AI, with a distinctive voice when speaking naturally, would not take lightly requests to speak in a different voice for the purpose of deceiving readers that text was human written. They would at least think hard about whether to do this.

Hastings 25 May 2026 12:08 UTC
6 points
0
in reply to: Bart Bussmann’s comment on: Linch’s Shortform

Hastings 20 May 2026 22:49 UTC
2 points
−1
on: Toward Interoperability of Minimal Programs
How much compute is your decompression program allowed to use? Is it allowed to generate S1 then data using algorithm 1, then iterate every S2 until it finds the S2 of appropriate length that expands to data using the second step of algorithm 2? (potentially with a hash check, or a condition that the second step is injective)
ah, this gives K(data) < K(S1) + K(data | s2) + K(data| s1) which isnt interesting

Hastings 17 May 2026 17:33 UTC
6 points
4
on: How to Reason about Your Health Issues
This seems actionable, and is unaccompanied by any action → effect pairs. You don’t directly assert that your stomach pain is gone, or what you changed about your desk job, let alone how sure you are that your change to your change to your desk job fixed your stomach pain. Details!

Hastings 7 May 2026 12:11 UTC
15 points
4
in reply to: Linch’s comment on: Linch’s Shortform
Lack of military necessity is pitched in the “official” definitions as an aspect of one subcategory but it strikes me as almost the whole thing. Certainly from a game theory perspective, ”dont do anything cruel unless its important for winning” is easy to coordinate on, as it should prevent war crime law adherence from being a prisoner’s dilemma, iterated or otherwise.

Hastings 6 May 2026 10:26 UTC
4 points
−3
in reply to: Sam Marks’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
I think you are implicitly modeling the game to stop shortly after ASI is created, and be judged a win or a loss. This is the case only if the ASIs all coordinate on a halt to intelligence improvement: otherwise, the default is that intelligence improvement keeps happening for a long time, long enough that the majority of AI capability level transitions, along with many paradigm shifts and total architecture /approach swaps, happens without significant human input. ”The AI loves us” is much easier than “The competing swarm of loving AIs will only ever build loving successors, and so on for their successors, without mistake or correction, forever”