johnswentworth

Karma: 65,162

johnswentworth 18 Sep 2026 15:01 UTC
5 points
2
in reply to: gzjw’s comment on: johnswentworth’s Shortform
Hope I haven’t misunderstood—you’re looking for some high-quality critiques of rationalism.
Nope! People ask for those all the time, that horse has already been thoroughly beaten. I am looking for the lowest of low-quality critiques.

johnswentworth 18 Sep 2026 2:21 UTC
10 points
3
on: johnswentworth’s Shortform
New bounty: $10 to write a “So You Want To Write A Hitpiece On The Rationalists...” guide-post on Lesswrong, in order to provide helpful material for all the people writing hit pieces lately.
It would just be such a shame if the rationalist community ~~got less free press~~ were not brought to sufficient justice just because lazy researchers didn’t find the juiciest and most scandalous tidbits.
If there are multiple entries then the $10 prize will go to my favorite. If you are tempted but don’t want to hunt down citations for all your juicy tidbits, it is acceptable to refer readers to David Gerard for all references.

johnswentworth 17 Sep 2026 17:56 UTC
2 points
1
in reply to: Boris Kjær’s comment on: Toward Interoperability of Minimal Programs
Fixed, thanks!

johnswentworth 17 Sep 2026 16:43 UTC
25 points
23
in reply to: mattmacdermott’s comment on: Thomas Kwa’s Shortform
Kinda yes, kinda no.
It is, in some sense, very blatantly obvious that sufficiently capable AI trained via ~any kind of RL will display the kinds of problems we saw in the huggingface incident. It is very blatantly obvious that nobody has any idea how to align highly capable models (other than crossing our fingers and hoping we get lucky), or even how to verify alignment. Etc. If one has already grasped these blatantly obvious facts, then the huggingface incident offers basically no information about how dangerous much stronger AI will be. (I personally have not updated at all about how dangerous much stronger AI will be as a result of the huggingface incident, and I think that is correct.)
… but to put it politely, some Bayesians start from impressively terrible priors. Some people need to be hit over the head really hard by empirical evidence in order to get such things. For people who needed to be hit over the head that hard, huggingface is an update, and it is correct to point it out as such.

johnswentworth 17 Sep 2026 1:50 UTC
4 points
0
in reply to: jacquesthibs’s comment on: Thomas Kwa’s Shortform
Either.

johnswentworth 17 Sep 2026 1:47 UTC
18 points
9
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
The basic thing one would want to do with that system is fully reverse engineer it in order to confirm that one’s understanding of general intelligence is (a) correct, in the sense that the system does not falsify any of it, and (b) complete, in the sense that one’s models fully account for the relevant aspects of the system’s behavior/internals.
… of course today we do not have such understanding of general intelligence to check against the system, so in practice one would also need to develop the understanding/models in parallel with reverse engineering the system.
And that would dramatically lower my p(doom).

johnswentworth 16 Sep 2026 22:50 UTC
39 points
22
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
My knee-jerk reaction is “no, current or near-future models aren’t the models whose alignment determines whether we live or die, and the alignment of current or near-future models doesn’t tell me much about alignment of the models which determine whether we live or die, so concrete empirical achievements on current or near-future models just aren’t very relevant”.
Reflecting for a bit, I guess if some empirical results made a really compelling case that current models aren’t that capable and near-future models won’t be either, that would send my p(doom) down a lot? Like e.g. if it turned out that ML had a 2010′s-psychology-level replication crisis and it turned out that nearly everything was fake, the Navier Stokes blowup problem was not actually solved by LLMs, etc. Not sure if my p(doom) would go all the way down to 25%, but it’d be close.

johnswentworth 9 Sep 2026 15:11 UTC
1 point
0
in reply to: davekasten’s comment on: Contra Piper on When Conversation Is Possible
I was not claiming that Kelsey Piper in particular feels like her self-models are sacred or inviolable, or that she gets super triggered when people question them. I rarely read Kelsey Piper, I don’t have much of a take on her at all.

johnswentworth 8 Sep 2026 20:22 UTC
8 points
2
in reply to: Ben Pace’s comment on: Contra Piper on When Conversation Is Possible
I suspect the people you’re referring to are concerned that they basically will have their voice silenced as a result...
I don’t know who you think I’m subtweeting here but I’m pretty sure I am not subtweeting them. For instance, the actual fights I’ve had about this have been with romantic partners (it has come up with more than one of them).
… anyway, I dunno about you, but if A gets triggered by B questioning A’s self-model, that’s one of the biggest updates for me that A has wildly inaccurate self-models. After all, you won’t end up with an accurate self-model if the main model-error-correction mechanism is broken. If someone questions my own self-model, and I respond by calmly talking through the actual evidence out loud, that also serves as a costly signal that I think clearly about my own self-model.

johnswentworth 8 Sep 2026 14:52 UTC
9 points
2
in reply to: 307th’s comment on: Contra Piper on When Conversation Is Possible
I do agree that a lot of people just incompetently throw around hypotheses of others’ self-models being wrong. Trying to convince someone that they’re suffering from a bias is indeed usually pretty useless. But I do not think e.g. addressing their concrete reasons is typically helpful at all.
My go-to prototypical example is hangriness. When someone is hangry, it’s often more obvious from the outside. There is easy-to-notice evidence, like e.g. “did this person get angry about three completely unrelated things one after another?”. And when I notice someone is hangry, the correct response is not to argue against whatever they’re object-level angry about at the moment; even if I “succeed” at that argument, they’ll just immediately find a different thing to be angry about. The correct response is to go find food and put it in front of them.

johnswentworth 8 Sep 2026 5:41 UTC
30 points
21
on: Contra Piper on When Conversation Is Possible
I think there’s some kind of relatively binary culture split on this issue.
Many people (including me and apparently you) are like “duh, of course peoples’ self-models are wrong all the time, so of course we need to treat them like any other claims people make, i.e. use ordinary reasoning and evidence to determine whether they’re true”. (And more generally, beyond rationalist circles, many people just treat claims about peoples’ self-models the same way they treat any other claims.)
Many other people feel like peoples’ self-models are sacred and inviolable, and those people get SUPER triggered if person A claims that person B is wrong about person B’s own internal motivations, emotions, memories, etc. You can see an anonymous example on LessWrong here, for instance.
I have had big fights about this before with people close to me, and I have a post somewhere in the drafts pile attempting to cross the communication gap from each side.

johnswentworth 8 Sep 2026 5:18 UTC
39 points
20
on: Contra Piper on When Conversation Is Possible
It certainly doesn’t make conversation with me impossible. When someone tells me that my understanding of my beliefs and motives should be rejected, I don’t take my ball and go home in a huff, muttering that they’ve made further conversation nearly impossible. Rather, I respond the same way I do to any other disagreement: by supporting my view with reasoning and evidence. Why would the rules be different just because the topic of discussion happens to be my own beliefs and motives?
Man, I was right there with you going “Yes! That! Exactly!” right up until “I respond [...] by supporting my view with reasoning and evidence”. There is a very important step before that! In cases where I am wrong about my own beliefs or motives, it is very difficult to notice that myself; other people pointing it out is one of the very few ways to notice. So when someone suggests I am wrong about my beliefs or motives, is extra super-duper important to step back, check the evidence, and consider whether they might be right!
And of course they could turn out right or wrong, I need to use reasoning and evidence to figure that out just like I do for everything else. The rules are indeed not any different just because the topic of discussion happens to be my own internal mental states and mechanisms.

johnswentworth 7 Sep 2026 16:24 UTC
19 points
7
on: Preliminary thoughts on Arrogance of the Humbled
I can talk a bit about how I distinguish “the problem is difficult” vs “I suck at the problem”.
Calibration on problems across the difficulty spectrum
Problems of different difficulty feel different.
I have spent at least a month of full time effort each directly tackling P vs NP, large integer factorization, and the Collatz conjecture. All of those are problems which enough very-smart people have spent enough time on that I can be confident they’re hard for the rest of humanity, not just for me. Having done that, I know what those sorts of problems qualitatively feel like.
I have also spent at least a month of full time effort each studying human aging and interp on a vision net. These are problems which are unsolved, but which people do make unambiguous progress on from time to time. And they feel qualitatively different from hard problems like P vs NP or Collatz.
I have also spent at least a month of full time effort each on proving a theorem about stability of a class of biological signalling systems and on building a cold-start recommendation system for social events. Each of these were hard enough that the first try didn’t work well, the first ten tries didn’t work well, I had to find some nontrivial new-to-me insights along the way to making them work. But they were tractable, and I eventually solved them very well. So I know what that difficulty level feels like.
And of course I have spent many months in school on homework problems and projects which require some work but are ultimately straightforward. Anyone with a technical degree probably knows what that feels like.
By working on problems which span the difficulty spectrum, I have a calibrated intuitive sense of the difficulty of different problems. I have some idea of what it feels like when I’m just not smart enough, vs what it feels like when no human out there is smart enough.
Know All Of The Things
The other thing which helps to distinguish “the problem is difficult” vs “I suck at the problem” is to have already studied every field and subject which seems at all relevant to the problem. I have studied a lot of material across a lot of fields. Every time someone comes along and says “maybe <subfield> has relevant ideas”, I at least look into <subfield> enough to form my own judgement of whether it has relevant ideas, and if so I study it. I have read through multiple course catalogs cover-to-cover, and I have read through every title in several years’ archives of Nature and of certain Arxiv tags, in search of any relevant obscure subfield I might be missing.
Bad News and Good News
Both of those methods—calibration on problems of various difficulty and studying everything at all relevant—take a lot of time and effort. They are not cheap. They are not tricks you can just whip out next time you’re wondering whether the problem is difficult or you suck.
The good news is that both of them are capital investments. The cost is mostly one-time, upfront cost. It’s years worth of upfront cost, but once it’s paid, you can keep reusing the things you’ve learned cheaply in-the-moment for a long time to come.

johnswentworth 4 Sep 2026 21:49 UTC
12 points
5
in reply to: Seth Herd’s comment on: Almost nobody is funded to figure out what work would solve alignment
You and others think the hard parts are apparent.
I don’t, actually. I don’t think anyone has ever even come close to conveying the hard problems to people who do not already have at least some sense of them. A few write-ups that I think made it farther across the gap than usual include Eliezer’s List O’Doom, the Embedded Agents sequence, and my own What Is The Alignment Problem? and The Plan posts. But none of those really get the whole picture across. And even before people are ready for those, most people need to pick up at least some simple prerequisite concepts of the sort The Sequences covered, though different people are missing different prerequisites and we also don’t have a single source which covers all the prerequisites.
So my actual take here is not exactly “don’t fund people writing about the hard problems of alignment”. If e.g. your plan was to approach people who already seem to demonstrate substantial understanding of the hard parts of the problem and offer them money to write a few posts posts trying to convey marginally more, I’d say that’s a reasonable plan. On the other hand, I would not recommend trying to e.g. hire newly minted MATS scholars to do this sort of thing. It is important, but decidedly not one of the more tractable directions.

johnswentworth 4 Sep 2026 17:11 UTC
55 points
19
on: Almost nobody is funded to figure out what work would solve alignment
I think this is missing the main reason to not devote more resources to the meta-problem.
One of the main patterns in alignment work is that people run into the hard parts of the problem, where it’s hard to even think clearly at all or find any hint of traction, and then they mentally “slide off” to something which feels easier. I claim that “go meta” is one of the most common ways that people slide off. Problem is, the meta problem is not actually easier than the object-level problem. It feels easier, because it’s further removed from concrete challenges and therefore easier to not-notice the ways in which one’s vague abstract ideas don’t match reality. But in actuality, if someone is struggling to even keep their attention on the object-level problems, they’re not going to be able to make actually-useful progress on the meta problem.

johnswentworth 3 Sep 2026 23:52 UTC
13 points
8
in reply to: cousin_it’s comment on: Cat-Belling Problems
Seems too narrow. For instance, the second law of thermodynamics isn’t a conservation law. Nonetheless, if somebody shows me a design for a type-2 perpetual motion machine, it makes sense to ask where precisely entropy goes down.

johnswentworth 3 Sep 2026 14:24 UTC
2 points
0
in reply to: Elizabeth’s comment on: johnswentworth’s Shortform
Yes, but nearly everyone who actually dies of them are frail 80-year-olds who were going to die of whatever metaphorical gust of wind hit them next. There are more human-specific infectious diseases now, but very few deaths from them, other than people who were on death’s door anyway.

johnswentworth 2 Sep 2026 17:43 UTC
62 points
0
on: johnswentworth’s Shortform
Astralcodexten has a new post on amygdalectomy, i.e. surgically eliminating the amygdala from the brain (ideally noninvasively, though for the guest author it happened incidentally in a much larger cancer operation). This was something David and I had previously speculated about, so I’m pretty excited to see any account of it, and even more excited about the big positive effects the guy saw.
My background mental model here is that humans have various physiological knobs to turn, and many of those are set by evolution to levels which are much worse in the modern environment than the ancestral environment. For instance:
- In the ancestral environment, infection was a top killer. Today it’s a rounding error. Our immune systems are therefore set by evolution to be far too sensitive for the modern environment.
- In the ancestral environment, food was much less abundant, so our bodies can use energy a lot more liberally today. In particular, this means thyroid levels are best on the high end of the normal range (but still not hyperthyroid); within the normal range, it’s a lot better to be generally high-energy than low-energy.
- The modern environment just generally has way less risk than the ancestral environment, and therefore humans’ general levels of anxiety, fear and stress are just ridiculously too high. This is where an amygdalectomy comes in: nuking the amygdala is the obvious way to permanently turn down the fear-and-anxiety knob.
The challenge, of course, is adjusting those knobs in a way that doesn’t cause a cascade of other problems.
Notably, the ACX post says (very tentatively) that the guy’s amygdala removal came with a pretty big IQ increase (7 points)! I’d be surprised if the effect is typically that large, but the IQ increase itself is a hypothesis I would have at least privileged beforehand. The mental model behind that part: in LLMs, people informally observe that RL training seems to make the LLMs stupider relative to their purely-predictive pretraining baseline. I suspect something similar applies to human minds. In both cases, there’s a baseline that’s basically objective-neutral, and insofar as there’s any general intelligence that’s where it lives. Then some hacky steering gets applied on top of that baseline. The hacky steering costs capability, but makes the system do specific things (e.g. talk like a chatbot or have babies), so it’s worth some capability tradeoff from the perspective of engineers/evolution. In humans, this looks like various hardcoded reward circuits—including many strong emotions—impairing clear thought. Remove one or two of those reward circuits, especially if they’re poorly adapted to the modern environment, and the human becomes more generally capable. I suspect that my own inability to feel love is one example, the amygdalectomy is another, and the general pattern of high-IQ people being somewhat autistic is a third.

johnswentworth 31 Aug 2026 15:26 UTC
13 points
8
in reply to: sarahconstantin’s comment on: sarahconstantin’s Shortform
EA orgs have a tendency to undervalue “ops” work, treating a variety of specialized tasks as though they should all be done by a usually female “office mom” whom they don’t really respect.
I don’t know about “the EA orgs”, but the rationalists closest to me are very explicit about the best ops people being some of the most important people around. Most of the Lightcone team are ops generalists. I have long advised would-be org founders that advice #1 is “don’t found an org”, #2 is “don’t grow the org”, and #3 is “if you really need to found an org, then your very first hire should be a super ops person”.
“A Message to Garcia” comes up now and then on LessWrong; it explains the ethos of the super ops person quite nicely. A little snippet from the beginning:
When war broke out between Spain and the United States, it was very necessary to communicate quickly with the leader of the Insurgents. Garcia was somewhere in the mountain fastnesses of Cuba—no one knew where. No mail or telegraph message could reach him. The President must secure his cooperation, and quickly.
What to do!
Someone said to the President, “There is a fellow by the name of Rowan [Lt. Andrew S. Rowan, U.S. Army] will find Garcia for you, if anybody can.”
Rowan was sent for and given a letter to be delivered to Garcia. How the “fellow by the name of Rowan” took the letter, sealed it up in an oilskin pouch, strapped it over his heart, in four days landed by night off the coast of Cuba from an open boat, disappeared into the jungle, and in three weeks came out on the other side of the Island, having traversed a hostile country on foot, and delivered his letter to Garcia—are things I have no special desire now to tell in detail. The point that I wish to make is this: McKinley gave Rowan a letter to be delivered to Garcia; Rowan took the letter and did not ask, “Where is he at?”
By the Eternal! there is a man whose form should be cast in deathless bronze and the statue placed in every college of the land. It is not book-learning young men need, nor instruction about this and that, but a stiffening of the vertebrae which will cause them to be loyal to a trust, to act promptly, concentrate their energies: do the thing—”Carry a message to Garcia.”
A simple test for a super ops person: if for some reason they literally needed to get a message to a guerilla leader in the mountains of Cuba, would they just go do that? Not necessarily alone, they might hire people, but would they ensure that the message reached its intended recipient, without org leadership having to pay any further attention to the matter? When I think of my shortlist of “super ops people” the answer is “yes” for every one of them; every one of them would get the message to Garcia.

johnswentworth 27 Aug 2026 14:43 UTC
16 points
5
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I’ll take a stab at this. I think Ben’s answer gestured at many of the right surface-level phenomena, but his summary of the underlying pattern is wrong.
People who culturally feel like “rationalists” moreso than “EAs” tend to have a deep distrust of mainstream status, “experts”, corporate ladder-climbing, simulacrum ³⁄₄ games, etc. You can see examples in:
- Zvi’s writing on Moral Mazes or simulacrum levels or his regular commentary on Very Serious People
- My own But It’s Fake Tho, Leading The Parade, or this comment from two days ago
- Eliezer’s writing on Inadequate Equilibria or modest epistemology or especially the Hero Licensing post
People who feel culturally more like “EAs”… don’t have that mistrust. Sometimes they’re just starry-eyed naive youngsters who haven’t really thought about these problems at all. Other times they’ll argue epistemic modesty, or (more often) say that playing along with the social dynamics is necessary in order to get big things done. (Some of the above links respond to those sorts of arguments.) Either way, the upshot is that more “EA” coded people are much willing to play the sort of social games which involve dishonesty, climbing social hierarchies, pandering to the meme mill, etc.

johnswentworth

Calibration on problems across the difficulty spectrum

Know All Of The Things

Bad News and Good News