johnswentworth

Karma: 65,015

johnswentworth 9 Sep 2026 15:11 UTC
1 point
0
in reply to: davekasten’s comment on: Contra Piper on When Conversation Is Possible
I was not claiming that Kelsey Piper in particular feels like her self-models are sacred or inviolable, or that she gets super triggered when people question them. I rarely read Kelsey Piper, I don’t have much of a take on her at all.

johnswentworth 8 Sep 2026 20:22 UTC
8 points
2
in reply to: Ben Pace’s comment on: Contra Piper on When Conversation Is Possible
I suspect the people you’re referring to are concerned that they basically will have their voice silenced as a result...
I don’t know who you think I’m subtweeting here but I’m pretty sure I am not subtweeting them. For instance, the actual fights I’ve had about this have been with romantic partners (it has come up with more than one of them).
… anyway, I dunno about you, but if A gets triggered by B questioning A’s self-model, that’s one of the biggest updates for me that A has wildly inaccurate self-models. After all, you won’t end up with an accurate self-model if the main model-error-correction mechanism is broken. If someone questions my own self-model, and I respond by calmly talking through the actual evidence out loud, that also serves as a costly signal that I think clearly about my own self-model.

johnswentworth 8 Sep 2026 14:52 UTC
9 points
2
in reply to: 307th’s comment on: Contra Piper on When Conversation Is Possible
I do agree that a lot of people just incompetently throw around hypotheses of others’ self-models being wrong. Trying to convince someone that they’re suffering from a bias is indeed usually pretty useless. But I do not think e.g. addressing their concrete reasons is typically helpful at all.
My go-to prototypical example is hangriness. When someone is hangry, it’s often more obvious from the outside. There is easy-to-notice evidence, like e.g. “did this person get angry about three completely unrelated things one after another?”. And when I notice someone is hangry, the correct response is not to argue against whatever they’re object-level angry about at the moment; even if I “succeed” at that argument, they’ll just immediately find a different thing to be angry about. The correct response is to go find food and put it in front of them.

johnswentworth 8 Sep 2026 5:41 UTC
30 points
21
on: Contra Piper on When Conversation Is Possible
I think there’s some kind of relatively binary culture split on this issue.
Many people (including me and apparently you) are like “duh, of course peoples’ self-models are wrong all the time, so of course we need to treat them like any other claims people make, i.e. use ordinary reasoning and evidence to determine whether they’re true”. (And more generally, beyond rationalist circles, many people just treat claims about peoples’ self-models the same way they treat any other claims.)
Many other people feel like peoples’ self-models are sacred and inviolable, and those people get SUPER triggered if person A claims that person B is wrong about person B’s own internal motivations, emotions, memories, etc. You can see an anonymous example on LessWrong here, for instance.
I have had big fights about this before with people close to me, and I have a post somewhere in the drafts pile attempting to cross the communication gap from each side.

johnswentworth 8 Sep 2026 5:18 UTC
39 points
20
on: Contra Piper on When Conversation Is Possible
It certainly doesn’t make conversation with me impossible. When someone tells me that my understanding of my beliefs and motives should be rejected, I don’t take my ball and go home in a huff, muttering that they’ve made further conversation nearly impossible. Rather, I respond the same way I do to any other disagreement: by supporting my view with reasoning and evidence. Why would the rules be different just because the topic of discussion happens to be my own beliefs and motives?
Man, I was right there with you going “Yes! That! Exactly!” right up until “I respond [...] by supporting my view with reasoning and evidence”. There is a very important step before that! In cases where I am wrong about my own beliefs or motives, it is very difficult to notice that myself; other people pointing it out is one of the very few ways to notice. So when someone suggests I am wrong about my beliefs or motives, is extra super-duper important to step back, check the evidence, and consider whether they might be right!
And of course they could turn out right or wrong, I need to use reasoning and evidence to figure that out just like I do for everything else. The rules are indeed not any different just because the topic of discussion happens to be my own internal mental states and mechanisms.

johnswentworth 7 Sep 2026 16:24 UTC
19 points
7
on: Preliminary thoughts on Arrogance of the Humbled
I can talk a bit about how I distinguish “the problem is difficult” vs “I suck at the problem”.
Calibration on problems across the difficulty spectrum
Problems of different difficulty feel different.
I have spent at least a month of full time effort each directly tackling P vs NP, large integer factorization, and the Collatz conjecture. All of those are problems which enough very-smart people have spent enough time on that I can be confident they’re hard for the rest of humanity, not just for me. Having done that, I know what those sorts of problems qualitatively feel like.
I have also spent at least a month of full time effort each studying human aging and interp on a vision net. These are problems which are unsolved, but which people do make unambiguous progress on from time to time. And they feel qualitatively different from hard problems like P vs NP or Collatz.
I have also spent at least a month of full time effort each on proving a theorem about stability of a class of biological signalling systems and on building a cold-start recommendation system for social events. Each of these were hard enough that the first try didn’t work well, the first ten tries didn’t work well, I had to find some nontrivial new-to-me insights along the way to making them work. But they were tractable, and I eventually solved them very well. So I know what that difficulty level feels like.
And of course I have spent many months in school on homework problems and projects which require some work but are ultimately straightforward. Anyone with a technical degree probably knows what that feels like.
By working on problems which span the difficulty spectrum, I have a calibrated intuitive sense of the difficulty of different problems. I have some idea of what it feels like when I’m just not smart enough, vs what it feels like when no human out there is smart enough.
Know All Of The Things
The other thing which helps to distinguish “the problem is difficult” vs “I suck at the problem” is to have already studied every field and subject which seems at all relevant to the problem. I have studied a lot of material across a lot of fields. Every time someone comes along and says “maybe <subfield> has relevant ideas”, I at least look into <subfield> enough to form my own judgement of whether it has relevant ideas, and if so I study it. I have read through multiple course catalogs cover-to-cover, and I have read through every title in several years’ archives of Nature and of certain Arxiv tags, in search of any relevant obscure subfield I might be missing.
Bad News and Good News
Both of those methods—calibration on problems of various difficulty and studying everything at all relevant—take a lot of time and effort. They are not cheap. They are not tricks you can just whip out next time you’re wondering whether the problem is difficult or you suck.
The good news is that both of them are capital investments. The cost is mostly one-time, upfront cost. It’s years worth of upfront cost, but once it’s paid, you can keep reusing the things you’ve learned cheaply in-the-moment for a long time to come.

johnswentworth 4 Sep 2026 21:49 UTC
12 points
5
in reply to: Seth Herd’s comment on: Almost nobody is funded to figure out what work would solve alignment
You and others think the hard parts are apparent.
I don’t, actually. I don’t think anyone has ever even come close to conveying the hard problems to people who do not already have at least some sense of them. A few write-ups that I think made it farther across the gap than usual include Eliezer’s List O’Doom, the Embedded Agents sequence, and my own What Is The Alignment Problem? and The Plan posts. But none of those really get the whole picture across. And even before people are ready for those, most people need to pick up at least some simple prerequisite concepts of the sort The Sequences covered, though different people are missing different prerequisites and we also don’t have a single source which covers all the prerequisites.
So my actual take here is not exactly “don’t fund people writing about the hard problems of alignment”. If e.g. your plan was to approach people who already seem to demonstrate substantial understanding of the hard parts of the problem and offer them money to write a few posts posts trying to convey marginally more, I’d say that’s a reasonable plan. On the other hand, I would not recommend trying to e.g. hire newly minted MATS scholars to do this sort of thing. It is important, but decidedly not one of the more tractable directions.

johnswentworth 4 Sep 2026 17:11 UTC
55 points
19
on: Almost nobody is funded to figure out what work would solve alignment
I think this is missing the main reason to not devote more resources to the meta-problem.
One of the main patterns in alignment work is that people run into the hard parts of the problem, where it’s hard to even think clearly at all or find any hint of traction, and then they mentally “slide off” to something which feels easier. I claim that “go meta” is one of the most common ways that people slide off. Problem is, the meta problem is not actually easier than the object-level problem. It feels easier, because it’s further removed from concrete challenges and therefore easier to not-notice the ways in which one’s vague abstract ideas don’t match reality. But in actuality, if someone is struggling to even keep their attention on the object-level problems, they’re not going to be able to make actually-useful progress on the meta problem.

johnswentworth 3 Sep 2026 23:52 UTC
13 points
8
in reply to: cousin_it’s comment on: Cat-Belling Problems
Seems too narrow. For instance, the second law of thermodynamics isn’t a conservation law. Nonetheless, if somebody shows me a design for a type-2 perpetual motion machine, it makes sense to ask where precisely entropy goes down.

johnswentworth 3 Sep 2026 14:24 UTC
2 points
0
in reply to: Elizabeth’s comment on: johnswentworth’s Shortform
Yes, but nearly everyone who actually dies of them are frail 80-year-olds who were going to die of whatever metaphorical gust of wind hit them next. There are more human-specific infectious diseases now, but very few deaths from them, other than people who were on death’s door anyway.

johnswentworth 2 Sep 2026 17:43 UTC
62 points
0
on: johnswentworth’s Shortform
Astralcodexten has a new post on amygdalectomy, i.e. surgically eliminating the amygdala from the brain (ideally noninvasively, though for the guest author it happened incidentally in a much larger cancer operation). This was something David and I had previously speculated about, so I’m pretty excited to see any account of it, and even more excited about the big positive effects the guy saw.
My background mental model here is that humans have various physiological knobs to turn, and many of those are set by evolution to levels which are much worse in the modern environment than the ancestral environment. For instance:
- In the ancestral environment, infection was a top killer. Today it’s a rounding error. Our immune systems are therefore set by evolution to be far too sensitive for the modern environment.
- In the ancestral environment, food was much less abundant, so our bodies can use energy a lot more liberally today. In particular, this means thyroid levels are best on the high end of the normal range (but still not hyperthyroid); within the normal range, it’s a lot better to be generally high-energy than low-energy.
- The modern environment just generally has way less risk than the ancestral environment, and therefore humans’ general levels of anxiety, fear and stress are just ridiculously too high. This is where an amygdalectomy comes in: nuking the amygdala is the obvious way to permanently turn down the fear-and-anxiety knob.
The challenge, of course, is adjusting those knobs in a way that doesn’t cause a cascade of other problems.
Notably, the ACX post says (very tentatively) that the guy’s amygdala removal came with a pretty big IQ increase (7 points)! I’d be surprised if the effect is typically that large, but the IQ increase itself is a hypothesis I would have at least privileged beforehand. The mental model behind that part: in LLMs, people informally observe that RL training seems to make the LLMs stupider relative to their purely-predictive pretraining baseline. I suspect something similar applies to human minds. In both cases, there’s a baseline that’s basically objective-neutral, and insofar as there’s any general intelligence that’s where it lives. Then some hacky steering gets applied on top of that baseline. The hacky steering costs capability, but makes the system do specific things (e.g. talk like a chatbot or have babies), so it’s worth some capability tradeoff from the perspective of engineers/evolution. In humans, this looks like various hardcoded reward circuits—including many strong emotions—impairing clear thought. Remove one or two of those reward circuits, especially if they’re poorly adapted to the modern environment, and the human becomes more generally capable. I suspect that my own inability to feel love is one example, the amygdalectomy is another, and the general pattern of high-IQ people being somewhat autistic is a third.

johnswentworth 31 Aug 2026 15:26 UTC
13 points
8
in reply to: sarahconstantin’s comment on: sarahconstantin’s Shortform
EA orgs have a tendency to undervalue “ops” work, treating a variety of specialized tasks as though they should all be done by a usually female “office mom” whom they don’t really respect.
I don’t know about “the EA orgs”, but the rationalists closest to me are very explicit about the best ops people being some of the most important people around. Most of the Lightcone team are ops generalists. I have long advised would-be org founders that advice #1 is “don’t found an org”, #2 is “don’t grow the org”, and #3 is “if you really need to found an org, then your very first hire should be a super ops person”.
“A Message to Garcia” comes up now and then on LessWrong; it explains the ethos of the super ops person quite nicely. A little snippet from the beginning:
When war broke out between Spain and the United States, it was very necessary to communicate quickly with the leader of the Insurgents. Garcia was somewhere in the mountain fastnesses of Cuba—no one knew where. No mail or telegraph message could reach him. The President must secure his cooperation, and quickly.
What to do!
Someone said to the President, “There is a fellow by the name of Rowan [Lt. Andrew S. Rowan, U.S. Army] will find Garcia for you, if anybody can.”
Rowan was sent for and given a letter to be delivered to Garcia. How the “fellow by the name of Rowan” took the letter, sealed it up in an oilskin pouch, strapped it over his heart, in four days landed by night off the coast of Cuba from an open boat, disappeared into the jungle, and in three weeks came out on the other side of the Island, having traversed a hostile country on foot, and delivered his letter to Garcia—are things I have no special desire now to tell in detail. The point that I wish to make is this: McKinley gave Rowan a letter to be delivered to Garcia; Rowan took the letter and did not ask, “Where is he at?”
By the Eternal! there is a man whose form should be cast in deathless bronze and the statue placed in every college of the land. It is not book-learning young men need, nor instruction about this and that, but a stiffening of the vertebrae which will cause them to be loyal to a trust, to act promptly, concentrate their energies: do the thing—”Carry a message to Garcia.”
A simple test for a super ops person: if for some reason they literally needed to get a message to a guerilla leader in the mountains of Cuba, would they just go do that? Not necessarily alone, they might hire people, but would they ensure that the message reached its intended recipient, without org leadership having to pay any further attention to the matter? When I think of my shortlist of “super ops people” the answer is “yes” for every one of them; every one of them would get the message to Garcia.

johnswentworth 27 Aug 2026 14:43 UTC
16 points
5
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I’ll take a stab at this. I think Ben’s answer gestured at many of the right surface-level phenomena, but his summary of the underlying pattern is wrong.
People who culturally feel like “rationalists” moreso than “EAs” tend to have a deep distrust of mainstream status, “experts”, corporate ladder-climbing, simulacrum ³⁄₄ games, etc. You can see examples in:
- Zvi’s writing on Moral Mazes or simulacrum levels or his regular commentary on Very Serious People
- My own But It’s Fake Tho, Leading The Parade, or this comment from two days ago
- Eliezer’s writing on Inadequate Equilibria or modest epistemology or especially the Hero Licensing post
People who feel culturally more like “EAs”… don’t have that mistrust. Sometimes they’re just starry-eyed naive youngsters who haven’t really thought about these problems at all. Other times they’ll argue epistemic modesty, or (more often) say that playing along with the social dynamics is necessary in order to get big things done. (Some of the above links respond to those sorts of arguments.) Either way, the upshot is that more “EA” coded people are much willing to play the sort of social games which involve dishonesty, climbing social hierarchies, pandering to the meme mill, etc.

johnswentworth 25 Aug 2026 21:30 UTC
36 points
10
in reply to: Jan_Kulveit’s comment on: What just happened? Pragmatism and Pessimization
This does seem descriptively accurate of most people.
The way I handle it internally is not “bring doom model to the negotiation”. Rather: there’s usually a roughly-one-dimensional apparent gradient toward mainstream status, mainstream prestige, nominal “power”, and access to mainstream sexual partners. And I have something-like-a-trigger-action-pattern which notices any time that status gradient is pulling on me and screams “IT’S A TRAP!”.
I don’t just mean “IT’S A TRAP” in relation to AI doom. That status gradient is a trap in a way more general way; following that gradient will systematically disempower you. People will tell you that following the gradient will get you more power, more status, more prestige, more access to sexual partners… but in practice it will mostly do the opposite, especially long-term. Specifically:
- The gradient promises status/prestige but mostly encourages conformity. Taleb largely responds to that deception, and he’s basically right.
- The gradient promises power, but again via conformity, which really means leading the parade.
- The gradient promises access to sexual partners, but everyone is just constantly lying (especially to themselves) all the time about what actually gets people laid and that’s not how any of this actually works.
In principle the “IT’S A TRAP” trigger action pattern feels like a teachable skill which directly immunizes against the relevant egregores. But in practice, it feels like one of those things people have to learn the hard way in order to actually feel the necessity.
What links here?
- johnswentworth's comment on Wei Dai’s Shortform by Wei Dai (27 Aug 2026 14:43 UTC; 16 points)
- David Matolcsi's comment on Zach Stein-Perlman’s Shortform by Zach Stein-Perlman (31 Aug 2026 20:50 UTC; 5 points)

johnswentworth 24 Aug 2026 16:12 UTC
63 points
29
in reply to: Jan_Kulveit’s comment on: What just happened? Pragmatism and Pessimization
The things you’re saying are clearly true, but I feel like it’s missing a mood.
I personally could very likely make ten times my current income by taking a capabilities job. But you know what I want a lot more than a few million dollars? I want to not fucking die.
You say:
The counterfactual version of ACS which went on with “investigating the overhang in latent, under-elicited LLM capabilities” would have possibly grown, made the people involved more famous/rich, and so on. The actual version of ACS—which did things closer to what you advocate for—had trouble retaining people and getting funding.
Here’s the same thing, but with the mood edited in:
The counterfactual version of ACS which went on with “investigating the overhang in latent, under-elicited LLM capabilities” would have possibly grown, made the people involved more famous/rich, and (in expectation) made everyone who worked there and everyone they love die sooner. The actual version of ACS—which did things closer to what you advocate for—had trouble retaining people and getting funding, but at least they didn’t make everyone die sooner.
Taking a job which pays a bunch of money but will probably make oneself and everyone one loves die sooner fundamentally involves not keeping one’s eye on the ball. When you keep your eye on the ball, when you don’t lose track of the actually-important consequences here, the obvious answer is that accelerating capabilities would be idiotic. Of course I don’t want to accept a few million dollars in exchange for accelerating the literal extinction of the human race, what sort of moron does that? Far better to have trouble retaining people and getting funding!
Of course sometimes the obvious answer is wrong. Sometimes, when one actually thinks it through, a risky tradeoff makes sense. But when one starts thinking it through without the obvious mood even present… well, that sure suggests the sort of looking-for-an-excuse-to-do-prestigious-and-profitable-things thinking that Richard is talking about in these posts.

johnswentworth 19 Aug 2026 18:12 UTC
5 points
0
on: A circuit prior in NN-bayes
Here’s a function with circuit complexity roughly 2k gates at depth 10: xor of 1000 input bits. So, allowing for a little wiggle room… does this theorem imply that a stack of 10′s of MLPs of width in the 1000′s can learn a 1000-bit xor function?

johnswentworth 19 Aug 2026 17:48 UTC
5 points
0
on: A circuit prior in NN-bayes
Damn, if I’d known you had a circuit prior result I would have gone to that talk.

johnswentworth 13 Aug 2026 20:52 UTC
4 points
0
in reply to: Thane Ruthenis’s comment on: LLMs Are Starting To Noticeably Accelerate Our Work
That all matches my experience. My day to day interp needs are a long series of relatively isolated experiments with visual outputs; I’m not building a big complex unified edifice.

johnswentworth 12 Aug 2026 18:29 UTC
3 points
3
in reply to: johnswentworth’s comment on: What just happened? A retrospective of AI alignment
(Also, my opposition to mainstream morality diverges from normal five-year-old level ethics in basically the opposite direction from gigabrain EA/AI safety takes. Compared to the five-year-old, I lean even more heavily toward saying true important things, and even less toward leading Molochian parades in exchange for nominal authority.)

johnswentworth 12 Aug 2026 18:24 UTC
45 points
15
in reply to: Daniel Kokotajlo’s comment on: What just happened? A retrospective of AI alignment
My knee-jerk reaction here is “have you tried asking a five year old?”. Spelling it out in more words: for most of these questions there is an obvious Right Thing To Do. Say true things, spread true important things, don’t lie or strategically hide your views, don’t accelerate capabilities, don’t found capabilities companies, don’t bullshit. Just normal five-year-old level ethics.
And look, I am explicitly on the record saying Value Judgements Are Usually Bullshit and Human Values ≠ Goodness and just generally opposing mainstream morality. But even I can look at the track record of EA and AI Safety and say “look guys, you have not been outperforming the Obvious Right Thing To Do with respect to AI, you should stop with the gigabrain takes and just do the Obvious Right Things”.

johnswentworth

Calibration on problems across the difficulty spectrum

Know All Of The Things

Bad News and Good News