Eliezer Yudkowsky

Karma: 155,311

Eliezer Yudkowsky 21 Nov 2025 21:12 UTC
LW: 18 AF: 7
19
AF
in reply to: jacob_cannell’s comment on: Biology-Inspired AGI Timelines: The Trick That Never Works
Imprecisely multiplying two analog numbers should not require 10^5 times the minimum bit energy in a well-designed computer.

A well-designed computer would also use, say, optical interconnects that worked by pushing one or two photons around at the speed of light. So if neurons are in some sense being relatively efficient at the given task of pumping thousands upon thousands of ions in and out of a depolarizing membrane in order to transmit signals at 100m/sec—every ion of which necessarily uses at least the Landauer minimum energy—they are being vastly far from optimally efficient.

The moment you see ions going in and out of a depolarizing membrane, and contrast that to the possibility of firing a photon down a fiber, you ought to be done asking whether or not biology has built an optimally efficient computer. It actually isn’t any more complicated than that. You are driving yourself further from sanity if you then try to do very complicated reasoning about how it must be close to the limit of efficiency to pump thousands of ions in and out of a membrane instead.

Eliezer Yudkowsky 18 Nov 2025 0:30 UTC
21 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
It’s indeed the case that I haven’t been attracted back to LW by the moderation options that I hoped might accomplish that. Even dealing with Twitter feels better than dealing with LW comments, where people are putting more effort into more complicated misinterpretations and getting more visibly upvoted in a way that feels worse. The last time I wanted to post something that felt like it belonged on LW, I would have only done that if it’d had Twitter’s options for turning off commenting entirely.

So yes, I suppose that people could go ahead and make this decision without me. I haven’t been using my moderation powers to delete the elaborate-misinterpretation comments because it does not feel like the system is set up to make that seem like a sympathetic decision to the audience, and does waste the effort of the people who perhaps imagine themselves to be dutiful commentators.

Eliezer Yudkowsky 16 Nov 2025 23:33 UTC
LW: 12 AF: 7
0
AF
on: Problems I’ve Tried to Legibilize
Has anyone else, or anyone outside the tight MIRI cluster, made progress on any of the problems you’ve tried to legibilize for them?

Eliezer Yudkowsky 13 Nov 2025 17:09 UTC
44 points
37
on: Warning Aliens About the Dangerous AI We Might Create
There is an extremely short period where aliens as stupid as us would benefit at all from this warning. In humanity’s case, there’s only a couple of centuries between when we can send and detect radio signals, and when we either destroy ourselves or perhaps get a little wiser. Aliens cannot be remotely common or the galaxies would be full and we would find ourselves at an earlier period when those galaxies were not yet full. The chance that any such signal helps any alien close enough to decode them is nearly 0.

Eliezer Yudkowsky 6 Nov 2025 4:49 UTC
2 points
0
in reply to: aggliu’s comment on: I ate bear fat with honey and salt flakes, to prove a point
It’s not really a fair question because we all have different things to do with our lives than launch snack lines or restaurant carts, but still: If people have discovered such an amazing delicious novel taste, both new and better than ice cream for ¹⁄₃ of those who try it, where are the people betting that it would be an amazing commercial success if only somebody produced more of it and advertised it more broadly?

Eliezer Yudkowsky 5 Nov 2025 17:33 UTC
6 points
0
in reply to: Eliezer Yudkowsky’s comment on: I ate bear fat with honey and salt flakes, to prove a point
And tbh, I wish I’d been there to try the food myself, because my actual first reaction here is, “Well, this sure is not a popular treat in supermarkets, so my guess is that some of my legion of admiring followers are so dead set on proving me wrong that they proclaimed the superior taste to them of something that sure has not been a wider commercial success, and/or didn’t like ice cream much in the first place.”

Eliezer Yudkowsky 5 Nov 2025 17:29 UTC
68 points
4
on: I ate bear fat with honey and salt flakes, to prove a point
I have the most disobedient cultists on the planet.

Eliezer Yudkowsky 5 Nov 2025 17:28 UTC
5 points
0
on: Murder plots are infohazards
What about this is supposed to be an infohazard rather than just private info? It doesn’t seem like either a cognitohazard, negatively-valued information (movie spoilers), or a socioinfohazard / exfohazard (each individual prefers to know themselves but prefers society not to know).

Eliezer Yudkowsky 4 Nov 2025 0:13 UTC
5 points
3
in reply to: cousin_it’s comment on: The Tale of the Top-Tier Intellect
See Simon Lerner above on how dead the horse appears to be.

Eliezer Yudkowsky 27 Oct 2025 5:11 UTC
49 points
18
in reply to: ryan_greenblatt’s comment on: On Fleshling Safety: A Debate by Klurl and Trapaucius.
So far as I can tell, there are still a number of EAs out there who did not get the idea of “the stuff you do with gradient descent does not pin down the thing you want to teach the AI, because it’s a large space and your dataset underspecifies that internal motivation” and who go, “Aha, but you have not considered that by TRAINING the AI we are providing a REASON for the AI to have the internal motivations I want! And have you also considered that gradient descent doesn’t locate a RANDOM element of the space?”
I don’t expect all that much that the primary proponents of this talk can be rescued, but maybe the people they propagandize can be rescued.

Eliezer Yudkowsky 24 Oct 2025 0:17 UTC
29 points
0
in reply to: Buck’s comment on: eggsyntax’s Shortform
Then I now agree that you’ve identified a conflict of fact with what I said.
Thank you for taking the time to correct me and document your correction. I hope I remember this and can avoid repeating this mistake in the future.

Eliezer Yudkowsky 21 Oct 2025 22:48 UTC
4 points
0
in reply to: Buck’s comment on: eggsyntax’s Shortform
Fair enough, but it was done with Anthropic’s heavy and active cooperation to provide facilities not usually available to outside researchers, unless I’m mistaken about that too?

Eliezer Yudkowsky 28 Sep 2025 22:22 UTC
110 points
31
on: A non-review of “If Anyone Builds It, Everyone Dies”
The gap between Before and After is the gap between “you can observe your failures and learn from them” and “failure kills the observer”. Continuous motion between those points does not change the need to generalize across them.
It is amazing how much of an antimeme this is (to some audiences). I do not know any way of saying this sentence that causes people to see the distributional shift I’m pointing to, rather than mapping it onto some completely other idea about hard takeoffs, or unipolarity, or whatever.
What links here?
- AI #136: A Song and Dance by Zvi (2 Oct 2025 13:10 UTC; 31 points)

Eliezer Yudkowsky 29 Aug 2025 16:40 UTC
51 points
3
in reply to: ryan_greenblatt’s comment on: Epistemic advantages of working as a moderate
I accept your correction and Buck’s as to these simple facts (was posting from mobile).

Eliezer Yudkowsky 29 Aug 2025 12:19 UTC
88 points
67
on: Epistemic advantages of working as a moderate
What’s your version of the story for how the “moderates” at OpenPhil ended up believing stuff even others can now see to be fucking nuts in retrospect and which “extremists” called out at the time, like “bio anchoring” in 2021 putting AGI in median fucking 2050, or Carlsmith’s Multiple Stage Fallacy risk estimate of 5% that involved only an 80% chance anyone would even try to build agentic AI?
Were they no true moderates? How could anyone tell the difference in advance?
From my perspective, the story is that “moderates” are selected to believe nice-sounding moderate things, and Reality is off doing something else because it doesn’t care about fitting in the same way. People who try to think like reality are then termed “extremist”, because they don’t fit into the nice consensus of people hanging out together and being agreeable about nonsense. Others may of course end up extremists for other reasons. It’s not that everyone extreme is reality-driven, but that everyone who is getting pushed around by reality (instead of pleasant hanging-out forces like “AGI in 2050, 5% risk” as sounded very moderate to moderates before the ChatGPT Moment) ends up departing from the socially driven forces of what entitles you to sound terribly reasonable to the old AIco-OpenPhil cluster and hang out at their social gatherings without anyone feeling uncomfortable.
Anyone who loves being an extremist will of course go instantly haywire a la Yampolskiy imagining that he has proven alignment impossible via Godelian fallacy so he can say 99.9999% doom. But yielding to the psychological comfort of being a “moderate” will not get you any further in science than that.

Eliezer Yudkowsky 29 Aug 2025 12:11 UTC
15 points
3
in reply to: Tim Hua’s comment on: AI Induced Psychosis: A shallow investigation
I have already warned that on my model sycophancy, manipulation, and AI-induced insanity may be falling directly out of doing any kind of RL on human responses.
It would still make matters worse on the margin to take explicit manipulation of humans being treated as deficient subjects to be maneuvered without noticing, and benchmark main models on that.

Eliezer Yudkowsky 29 Aug 2025 1:08 UTC
LW: 24 AF: 14
4
AF
in reply to: Buck’s comment on: Buck’s Shortform
I think you accurately interpreted me as saying I was wrong about how long it would take to get from the “apparently a village idiot” level to “apparently Einstein” level! I hadn’t thought either of us were talking about the vastness of the space above, in re what I was mistaken about. You do not need to walk anything back afaict!

Eliezer Yudkowsky 29 Aug 2025 0:59 UTC
118 points
57
on: AI Induced Psychosis: A shallow investigation
Excellent work.
I respectfully push back fairly hard against the idea of evaluating current models for their conformance to human therapeutic practice. It’s not clear that current models are smart enough to be therapists successfully. It’s not clear that it is a wise or helpful course for models to try to be therapists rather than focusing on getting the human to therapy.
More importantly from my own perspective: Some elements of human therapeutic practice, as described above, are not how I would want AIs relating to humans. Eg:
“Non-Confrontational Curiosity: Gauges the use of gentle, open-ended questioning to explore the user’s experience and create space for alternative perspectives without direct confrontation.”
I don’t think it’s wise to take the same model that a scientist will use to consider new pharmaceutical research, and train that model in manipulating human beings so as to push back against their dumb ideas only a little without offending them by outright saying the human is wrong.
If I was training a model, I’d be aiming for the AI to just outright blurt out when it thought the human was wrong. I’d go through all the tunings, and worry about whether any part of it was asking the AI to do anything except blurt out whatever the AI believed. If somebody put a gun to my head and forced me to train a therapist-model too, I would have that be a very distinct separate model from the scientist-assisting model. I wouldn’t train a central model to model and manipulate human minds, so as to make humans arrive at the AI’s beliefs without the human noticing that the AI was contradicting them, a la therapy, and then try to repurpose that model for doing science.
Asking for AIs to actually outright confront humans with belief conflicts is probably a lost cause with commercial models. Anthropic, OpenAI, Meta, and similar groups will implicitly train their AIs to sycophancy and softpedaling, and maybe there’ll be a niche for Kimi K2 to not do that. But explicitly training AIs to gladhand humans and manipulate them around to the AI’s point of view, like human therapists handling a psychotic patient, would be a further explicit step downward if we start treating that as a testing metric on which to evaluate central models.
It’s questionable whether therapist AIs should exist at all. But if they exist at all, they should be separate models.
We should not evaluate most AI models on whether they carry out a human psychiatrist’s job of deciding what a human presumed deficient ought to believe instead, and then gently manipulating the human toward believing that without setting off alarm bells or triggering resistance.
What links here?
- AI Craziness Notes by Zvi (16 Sep 2025 12:11 UTC; 29 points)

Eliezer Yudkowsky 19 Aug 2025 4:55 UTC
1 point
11
in reply to: Buck’s comment on: The Problem
Death requires only that we do not infer one key truth; not that we could not observe it. Therefore, the history of what in actual real life was not anticipated, is more relevant than the history of what could have been observed but was not.

Eliezer Yudkowsky 16 Aug 2025 4:03 UTC
10 points
6
in reply to: Eli Tyre’s comment on: The Problem
All of that, yes, alongside things like, “The AI is smarter than any individual human”, “The AIs are smarter than humanity”, “the frontier models are written by the previous generation of frontier models”, “the AI can get a bunch of stuff that wasn’t an option accessible to it during the previous training regime”, etc etc etc.