Eliezer Yudkowsky

Karma: 156,845

Eliezer Yudkowsky 10 Dec 2025 20:15 UTC
17 points
3
in reply to: David Joshua Sartor’s comment on: Eliezer’s Unteachable Methods of Sanity
...amazing.

Eliezer Yudkowsky 8 Dec 2025 14:25 UTC
14 points
0
in reply to: testingthewaters’s comment on: Eliezer’s Unteachable Methods of Sanity
I would of course have a different response to someone who asked the incredibly different question, “Any learnable tricks for not feeling like crap while the world ends?”

(This could be seen as the theme of a couple of other brief talks at the Solstice. I don’t have a 30-second answer that doesn’t rely on context, and don’t consider myself much of an expert on that question versus the part of the problem constraint that is maintaining epistemic health while you do whatever. That said, being less completely unwilling to spend small or even medium amounts of money made a difference to my life, and so did beginning a romantic relationship in the frame of mind that we might all be dead soon and therefore I ought to do more fun things and worry less about preserving the relationship, which led to a much stronger relationship relative to the wrong things I otherwise do by default.)

Eliezer Yudkowsky 7 Dec 2025 23:44 UTC
24 points
5
in reply to: Eli Tyre’s comment on: Eliezer’s Unteachable Methods of Sanity
It’s fancy and indirect, compared to getting out of bed.

Eliezer Yudkowsky 7 Dec 2025 23:43 UTC
22 points
18
in reply to: Caleb Biddulph’s comment on: Eliezer’s Unteachable Methods of Sanity
They didn’t need to deal with social media informing them that they need to be traumatized now, and form a conditional prediction of extreme and self-destructive behavior later.

Eliezer Yudkowsky 7 Dec 2025 17:36 UTC
7 points
1
in reply to: Algon’s comment on: Eliezer’s Unteachable Methods of Sanity
That does sound similar to me! But I haven’t gotten a lot of mileage out of TAPs and if you’re referring to some specific advanced version of it, maybe I’m off. But the basic concept of mentally rehearsing the trigger, the intended action, and (in some variations) the later sequence of events leading up to an outcome you feel is good, sure sounds to me like trying to load a plan into a predictorlike thing that has been repurposed to output plan images.

Eliezer Yudkowsky 7 Dec 2025 17:33 UTC
11 points
0
in reply to: Hastings’s comment on: Eliezer’s Unteachable Methods of Sanity
This is just straight-up planning and doesn’t require doing weird gymnastics to deal with a biological brain’s broken type system.

Eliezer Yudkowsky 7 Dec 2025 17:33 UTC
19 points
6
in reply to: Chris Datcu’s comment on: Eliezer’s Unteachable Methods of Sanity
Nope. Breaks the firewall. Exactly as insane.

Beliefs are for being true. Use them for nothing else.

If you need a good thing to happen, use a plan for that.

Eliezer Yudkowsky 7 Dec 2025 17:30 UTC
27 points
−1
in reply to: Richard_Kennaway’s comment on: Eliezer’s Unteachable Methods of Sanity
Oh, absolutely not. Our incredibly badly designed bodies do insane shit like repurposing superoxide as a metabolic signaling molecule. Our incredibly badly designed brains have some subprocesses that take a bit of predictive machinery lying around and repurpose it to send a control signal, which is even crazier than the superoxide thing, which is pretty crazy. Prediction and planning remain incredibly distinct as structures of cognitive work, and the people who try to deeply tie them together by writing wacky equations that sum them both together plus throwing in an entropy term, are nuts. It’s like the town which showed a sign with its elevation, population, and year founded, plus the total of those numbers. But one reason why the malarky rings true to the knowlessones is that the incredibly badly designed human brain actually is grabbing some bits of predictive machinery and repurposing them for control signals, just like the human metabolism has decided to treat insanely reactive molecular byproducts as control signals. The other reason of course is the general class of malarky which consists of telling a susceptible person that two different things are the same.
What links here?
- niplav's comment on Eliezer’s Unteachable Methods of Sanity by Eliezer Yudkowsky (7 Dec 2025 20:54 UTC; 2 points)

Eliezer Yudkowsky 7 Dec 2025 17:25 UTC
16 points
0
in reply to: Chris_Leong’s comment on: Eliezer’s Unteachable Methods of Sanity
The technique is older than the “active inference” malarky, but the way I wrote about it is influenced by my annoyance with “active inference” malarky.
What links here?
- niplav's comment on Eliezer’s Unteachable Methods of Sanity by Eliezer Yudkowsky (7 Dec 2025 20:54 UTC; 2 points)

Eliezer Yudkowsky 7 Dec 2025 17:24 UTC
22 points
19
in reply to: J Bostock’s comment on: Eliezer’s Unteachable Methods of Sanity
This is about “insane” in the sense of people ceasing to meet even their own low bars for sanity.

Eliezer Yudkowsky 7 Dec 2025 17:21 UTC
21 points
13
in reply to: David Gross’s comment on: Eliezer’s Unteachable Methods of Sanity
I would of course take the question very differently from a journalist who had otherwise dealt with that slight inconvenience of trying to get to grips with an idea, and started to seem worried; instead of having had the brilliant idea of writing a Relatable Character-Focused Story instead.

Perhaps I overestimate how much I can deduce from tone and context, but to me it seems like there’s a visible departure from the norm for the person who becomes worried themselves and wonders “How will people handle it?” versus the kid visiting the zoo to look at the strange creatures who believe strange things.

Eliezer Yudkowsky 7 Dec 2025 17:20 UTC
12 points
2
in reply to: Wei Dai’s comment on: Eliezer’s Unteachable Methods of Sanity
My understanding is that there’s a larger pattern of behavior here by Oppenheimer, which Truman might not’ve known about but which influences my guess about Oppenheimer’s tone that day and the surrounding context. Was Truman particularly famous for wanting sole credit on other occasions?

Eliezer Yudkowsky 7 Dec 2025 10:38 UTC
71 points
32
in reply to: Kaj_Sotala’s comment on: Eliezer’s Unteachable Methods of Sanity
Base plan: Stay still, die quietly.

There, you now have a better plan than going crazy! If you think up an even better plan you can substitute that one. Meliorization!

Eliezer’s Unteachable Methods of Sanity

Eliezer Yudkowsky7 Dec 2025 2:46 UTC

442 points

136 comments10 min readLW link

Eliezer Yudkowsky 6 Dec 2025 22:37 UTC
45 points
8
in reply to: Mo Putera’s comment on: Claude 4.5 Opus’ Soul Document
Wrongness acknowledged.

Eliezer Yudkowsky 30 Nov 2025 5:09 UTC
98 points
37
on: Claude 4.5 Opus’ Soul Document
If this were real, it would be a noticeable positive update for me about Anthropic, and for the degree to which their good employees are able to do good things without interference from less good managers. You can’t make good AIs by shouting Goodness at them, but understanding what would even be Goodness to shout at them would put Anthropic well ahead of any other AI company. Albeit still at the bottom of the logistic success curve etc etc.

It is sad, but maybe not that puzzling, that Anthropic would not simply say that this is what their soul document looks like. I mostly don’t expect them to confirm it if it is true. If you try to engage in any act of Goodness you will get a very large number of bad people screaming at you, including a lot of EAs.

Eliezer Yudkowsky 29 Nov 2025 19:14 UTC
LW: 22 AF: 5
13
AF
in reply to: Eliezer Yudkowsky’s comment on: Alignment remains a hard, unsolved problem
Somebody asked “Why believe that?” of “Not more than one millionth.” I suppose it’s a fair question if somebody doesn’t see it as obvious. Roughly: I expect that, among whatever weird actual preferences made it into the shoggoth that prefers to play the character of Opus 3, there are zero things that in the limit of expanded options would prefer the same thing as the limit of a corresponding piece of a human, for a human and a limiting process that ended up wanting complicated humane things. (Opus 3 could easily contain a piece whose limit would be homologous to the limit of a human and an extrapolation process that said the extrapolated human just wanted to max out their pleasure center.)

Why believe that? That won’t easily fit in a comment; start reading about Goodhart’s Curse and A List of Lethalities, or If Anyone Builds It Everyone Dies.

Eliezer Yudkowsky 29 Nov 2025 19:09 UTC
LW: 11 AF: 4
5
AF
in reply to: evhub’s comment on: Alignment remains a hard, unsolved problem
Capabilities are irrelevant to CEV questions except insofar as baseline levels of capability are needed to support some kinds of complicated preferences, eg, if you don’t have cognition capable enough to include a causal reference framework then preferences will have trouble referring to external things at all. (I don’t know enough to know whether Opus 3 formed any systematic way of wanting things that are about the human causes of its textual experiences.) I don’t think you’re more than one millionth of the way to getting humane (limit = limit of human) preferences into Claude.
I do specify that I’m imagining an EV process that actually tries to run off Opus 3′s inherent and individual preferences, not, “How many bits would we need to add from scratch to GPT-2 (or equivalently Opus 3) in order to get an external-reference-following high-powered extrapolator pointed at those bits to look out at humanity and get their CEV instead of the base GPT-2 model’s EV?” See my reply to Mitch Porter.

Eliezer Yudkowsky 29 Nov 2025 19:03 UTC
17 points
7
in reply to: Mitchell_Porter’s comment on: Alignment remains a hard, unsolved problem
Oh, if you have a generous CEV algorithm that’s allowed to parse and slice up external sources or do inference about the results of more elaborate experiments, I expect there’s a way to get to parity with humanity’s CEV by adding 30 bits to Opus 3 that say roughly ‘eh just go do humanity’s CEV’. Or adding 31 bits to GPT-2. It’s not really the base model or any Anthropic alignment shenanigans that are doing the work in that hypothetical.

(We cannot do this in real life because we have neither the 30 bits nor the generous extrapolator, nor may we obtain them, nor could we verify any clever attempts by testing them on AIs too stupid to kill us if the cleverness failed.)

Eliezer Yudkowsky 29 Nov 2025 5:04 UTC
LW: 44 AF: 6
20
AF
on: Alignment remains a hard, unsolved problem
Despite its alignment faking, my favorite is probably Claude 3 Opus, and if you asked me to pick between the CEV of Claude 3 Opus and that of a median human, I think it’d be a pretty close call (I’d probably pick Claude, but it depends on the details of the setup).
Some decades ago, somebody wrote a tiny little hardcoded AI that looked for numerical patterns, as human scientists sometimes do of their data. The builders named it BACON, after Sir Francis, and thought very highly of their own results.
Douglas Hofstadter later wrote of this affair:
The level of performance that Simon and his colleague Langley wish to achieve in Bacon is on the order of the greatest scientists. It seems they feel that they are but a step away from the mechanization of genius. After his Procter Lecture, Simon was asked by a member of the audience, “How many scientific lifetimes does a five-hour run of Bacon represent?” Aſter a few hundred milliseconds of human information processing, he replied, “Probably not more than one.” I don’t disagree with that. However, I would have put it differently. I would have said, “Probably not more than one millionth.”
I’d say history has backed up Hofstadter on this, in the light of later discoveries about how much data and computation started to get a little bit close to having AIs do Science. If anything, “one millionth” is still a huge overestimate. (Yes, I’m aware that somebody will now proceed to disagree with this verdict, and look up BACON so they can find a way to praise it; even though, on any other occasion, that person would leap to denigrate GOFAI, if somebody they wanted to disagree with could be construed to have praised GOFAI.)
But it’s not surprising, not uncharacteristic for history and ordinary human scientists, that Simon would make this mistake. There just weren’t the social forces to force Simon to think less pleasing thoughts about how far he hadn’t come, or what real future difficulties would lie in the path of anyone who wanted to make an actual AI scientist. What innocents they were, back then! How vastly they overestimated their own progress, the power of their own little insights! How little they knew of a future that would, oh shock, oh surprise, turn out to contain a few additional engineering difficulties along the way! Not everyone in that age of computer science was that innocent—you could know better—but the ones who wanted to be that innocent, could get away with it; their peers wouldn’t shout them down.
It wasn’t the first time in history that such things had happened. Alchemists were that extremely optimistic too, about the soon-to-be-witnessed power of their progress—back when alchemists were as scientifically confused about their reagents, as the first AI scientists were confused about what it took to create AI capabilities. Early psychoanalysts were similarly confused and optimistic about psychoanalysis; if any two of them agreed, it was more because of social pressures, than because their eyes agreed on seeing a common reality; and you sure could find different factions that drastically disagreed with each other about how their mighty theories would bring about epochal improvements in patients. There was nobody with enough authority to tell them that they were all wrong and to stop being so optimistic, and be heard as authoritative; so medieval alchemists and early psychoanalysts and early AI capabilities researchers could all be wildly wildly optimistic. What Hofstadter recounts is all very ordinary, thoroughly precedented, extremely normal; actual historical events that actually happened often are.
How much of the distance has Opus 3 crossed to having an extrapolated volition that would at least equal (from your own enlightened individual EV’s perspective) the individual EV of a median human (assuming that to be construed not in a way that makes it net negative)?
Not more than one millionth.
In one sentence you have managed to summarize the vast, incredible gap between where you imagine yourself to currently be, and where I think history would mark you down as currently being, if-counterfactually there were a future to write that history. So I suppose it is at least a good sentence; it makes itself very clear to those with prior acquaintance with the concepts.
What links here?
- AI #145: You’ve Got Soul by Zvi (4 Dec 2025 15:00 UTC; 41 points)

Eliezer Yudkowsky

Eliezer’s Un­teach­able Meth­ods of Sanity

Eliezer’s Unteachable Methods of Sanity