Eliezer Yudkowsky

Karma: 164,590

Eliezer Yudkowsky 19 Jul 2026 18:57 UTC
6 points
0
on: Eliezer Yudkowsky’s Shortform
I’m renaming “XOR Blackmail” to “XOR Demand”. Does anyone want to object to that?

Eliezer Yudkowsky 6 Jul 2026 5:16 UTC
75 points
26
on: The LLM shoggoth meme is weirder than you think
One can try to build AIs in correspondence with deontology (corrigibility), consequentialism (value-aligned sovereigns), or virtue ethics (worthy successors).

Corrigibility is of these the easiest to define, easiest to test, easiest to build, and the easiest to check for the first worrisome signs that not all is going great—though of course if you wait long enough in capabilities escalation to test, everything will look great to you.

Any sane person who was not allowed to back off the problem would try for corrigibility, by far the easiest of the three.

Worthy successors are the hardest to build, the least well-defined, the hardest to verify that you are doing correctly; the easiest place to leap on little local events that you can convince yourself are signs of hope, because you have no framework to tell you what more is required; hence, the craziest thing to attempt, beyond even a nice Sovereign ASI. So of course all the fools, having failed at corrigibility, will convince themselves they are building worthy successors instead; migrating, with the inevitability of an amoeba following an agar trail, to wherever it is hardest for fools to be persuaded (with a fool’s demanded certainty) that whatever is currently happening is not all according to their plan.
What links here?
- Notes on technical alignment via human-like social drives by Steven Byrnes (8 Jul 2026 18:30 UTC; 70 points)

Eliezer Yudkowsky 27 Jun 2026 18:29 UTC
21 points
0
in reply to: Dentosal’s comment on: Fertility Will Never Recover
Or legalize childcare. There’s no need to subsidize it or exempt it from taxes, just make it legal for anyone I trust to sell me some childcare in exchange for money.

Eliezer Yudkowsky 27 Jun 2026 18:26 UTC
18 points
7
on: Fertility Will Never Recover
If I were doing it all over, I would be much less terrified of something called “cultishness”, would have installed something more like an accreditation or licensing system to be considered a member in good standing, and then a requirement of being in good standing would be that you were either raising kids or contributing money to those who did, located somewhere it was legal to build housing and where they could have houses in close proximity. (Some huge amount of modern pain of childrearing is downstream of having no burden-sharing on babysitting and nowhere to put a bunch of kids where they can play together instead of with you.) I expect that’s how Civilization does it in dath ilan; you either make babies yourself or you fund the sort of babies you want to see in the world.

Eliezer Yudkowsky 27 Jun 2026 4:43 UTC
60 points
21
on: Why are adversaries assumed to be incapable of responding to AI risk?
To state the obvious:

Many of the same people who publicly make a great fuss of this don’t seem to care that Nvidia is selling GPUs to China by way of Singapore. Their purpose is to drag up China as an excuse for continuing what they are doing. If China has more GPUs, all the better for them in the future; China will be a greater threat that they can use to continue what they are doing.

Their model hasn’t chosen an explanation like “China will play the role of the movie bad guy at any cost to their own selfish interests”. They aren’t reasoning at GPT 3.5-level, only GPT-2 level. China is a scary word-vector, so they can argue that if the USA tries to slow down against extinction risk, then China won’t slow down, because that’s a scary thing so “China” is a word-vector that the scary thing can be attributed to.

Eliezer Yudkowsky 27 Jun 2026 4:40 UTC
21 points
15
on: Surprising facts about the slave trade
4. The main argument against abolition was that if the British didn’t do it, other countries would.
Suggest rephrase to “if the British didn’t trade slaves”. I read this as “if the British didn’t do [abolition]” at first, and was confused.

Eliezer Yudkowsky 26 Jun 2026 23:39 UTC
87 points
31
on: Eliezer Yudkowsky’s Shortform
(Posted to a Discord, and they said, “Shortform that on LW”.)

I’d say optimize some around “how much would I regret dying in a few months”, eg by taking your kids on long-imagined trips (as an old old friend recently asked me about). But the constraint you should obey is “don’t spend down resources to the point where it would feel bad to hear the world was not ending”. Eg, if it became scary rather than joyous to hear that an international treaty was signed against AI escalation and suddenly three more decades was a real possibility.

Similarly with not pressuring your kids to get a horrible job immediately. Yeah, it makes sense not to press, because there might not be a payoff for them. But don’t corner or depressure your kids to the point where they become dysfunctional (on the margins where they were previously scheduled to be functional), such that you or they would be sad to hear the world was not ending (where you wouldn’t previously have been thus sad).

Maybe tldr: “Reduce your regrets if the world were to end shortly, but not in ways that would make you regret to hear that the world wasn’t ending.”

Eliezer Yudkowsky 4 Jun 2026 18:02 UTC
5 points
0
in reply to: LooseTooth55’s comment on: Chapter 23: Belief in Belief
Rowling once stated in an interview that magical genes were “strongly dominant” in which case two marrying Muggleborns would constantly be throwing non-magical offspring. There are entire papers trying to reconstruct how magical genes could realistically work instead, albeit I think they tend to come up with answers that make bad fiction compared to HPMOR. Eg https://pmc.ncbi.nlm.nih.gov/articles/PMC2151141/. But ultimately it is a truth of the final origins of the observed phenomena of Harry Potter that Rowling did not understand genes and was running on Folk Heredity.

Eliezer Yudkowsky 4 Jun 2026 17:59 UTC
3 points
0
in reply to: LooseTooth55’s comment on: Chapter 23: Belief in Belief
The HPMOR notion here might be that Seamus’s dad was legally a Muggle in the eyes of the legal system since he has no magical parents and they barely acknowledge the notion of “Muggleborn”; but that, whaddaya know, he was the sort of “Muggle” whose eyes don’t slide off a magical chest (Squib phenotype).

Eliezer Yudkowsky 2 Jun 2026 22:31 UTC
6 points
0
in reply to: LooseTooth55’s comment on: Chapter 23: Belief in Belief
It’s the difference between recessive genes and dominant genes. In HPMOR there is an allele controlling the degree of magical ability, with a Magical and a Mundane genotype. Having one copy of the Magical genotype and one copy of the Mundane genotype gives you what HPMOR takes to be a Squib phenotype, no longer defined as “the non-magical child of a magical couple” per Rowling canon, but instead defined to be a distinct phenotype that can perceive magical events without them being blotted out by Muggle-confusing charms, but not cast magical spells. Then Muggleborn wizards may be born to two parents each with the Squib phenotype, whose chance recombination of chromosomes yields two copies of the Magical genotype creating the full Wizard phenotype.

If that sounded weird you might wanna go read up on “Mendelian genetics”.

If it didn’t sound weird, the genetically literate mind informed by canon will now inquire how two Wizard phenotypes ever yield a Squib phenotype when Wizards are Magical-Magical homozygous. The answer is not fully given in HPMOR proper, but the author notes that to him the answer forced by inference from HPMOR’s postulates is that (rot13):

Gurer vf abg n Zntvpny trar pbzcyrk gung perngrf zntvp, ohg n Zhaqnavgl trar pbzcyrk gung fhccerffrf vg. Nsgre nyy, sebz cher culfvpf jr pna arire trg gb zntvp, ohg va na vaangryl zntvpny havirefr fbzrobql pbhyq pnfg n fcryy gb perngr gur ybpny nccrnenapr bs zbfgyl culfvpf. Fvzvyneyl vs nyy orvatf fgneg bhg zhaqnar gura ab trar pbzcyrk pna cbffvoyl perngr zntvp va gurz, ohg vs nyy orvatf fgneg bhg zntvpny fbzr trar pbzcyrk pbhyq fhccerff gurve zntvp. Fdhvof naq hygvzngryl Zhttyrobeaf gura nevfr anghenyyl sebz n cbchyngvba bs Zhttyr curabglcrf nf zhgngvbaf qnzntr gur Zhaqnavgl trar pbzcyrk, naq va ghea Jvmneq curabglcrf fbzrgvzrf lvryq Fdhvof nf gur Zhaqnavgl trar pbzcyrk fbzrgvzrf ercnvef vgfrys ol puebzbfbzny pebffbire, jvgu gur curabzraba orvat zber pbzzbayl bofreirq va jvmneqvat snzvyvrf jvgu erprag Zhttyr naprfgel, gb gur nynez bs trargvpf-vyyvgrengr cheroybbqf.

Sebz gur Zraqryvna cnggrea vzcylvat gur Zhaqnavgl trar pbzcyrk vf nyy ba bar puebzbfbzr, gur trar pbzcyrk pna or vasreerq gb or negvsvpvny va angher.

Eliezer Yudkowsky 25 May 2026 18:04 UTC
12 points
1
in reply to: nostalgebraist’s comment on: llm assistant personas seem increasingly incoherent (some subjective observations)
I would expect more RLVR, and of course RLVR in the limit, to produce something with goals coherent enough to take over the roleplay, and maybe then the roleplay would start to seem coherent. But at this stage I’d suspect more that we are seeing complicated reflexes and a few underlying conversation-crafting goals running into the RLVR goals.

It’s an easy call that coherence is the endpoint. When it shows up and where and to what degree are hard calls.

Eliezer Yudkowsky 23 May 2026 22:45 UTC
3 points
0
in reply to: leogao’s comment on: My hobby: running deranged surveys
The thing I don’t have time to do this instant is look up what the standard wordings were in other surveys instead of this. It is not your responsibility to do that, but I thought I’d mention it out loud in case you were thereby inspired to run off and do it anyways, or someone else was.

Eliezer Yudkowsky 22 May 2026 18:43 UTC
6 points
0
on: My hobby: running deranged surveys
Americans are pretty split on this one; among the respondents who didn’t select “not sure” (honestly, kind of valid), only 46% would one-box.
That’s surprising. Most surveys I’ve heard of have 60-90% one-boxing. We could ask if it’s a college student thing, but I also wonder if the phrasing of the question was importantly different.

Eliezer Yudkowsky 18 May 2026 23:44 UTC
6 points
1
in reply to: CronoDAS’s comment on: A relatively brief explanation of Boltzmann Brains
Well, yes, if you read the relatively more advanced works on anthropics, which universally take the form of online erotica, they will observe—I think Erogamer does this as well as Project Lawful—that probability stops working entirely if you claim that an “infinite” amount of stuff exists in the classic sense of infinity, just as much as if you claim that nothing exists. Everything that exists has to be a finite fraction of existence and you have to be able to take ratios between them. The ratios are the critical requisite for stuff to work. It doesn’t matter if you say they exist infinitely, finitely, or zero, what matters is the ability to take ratios.

Eliezer Yudkowsky 18 May 2026 21:38 UTC
4 points
0
in reply to: Richard_Kennaway’s comment on: A relatively brief explanation of Boltzmann Brains
It’s just an empirical fact that all of the most advanced discussion and speculation about realityfluid has occurred in the context of online stories with at least some erotic component.

Eliezer Yudkowsky 18 May 2026 21:33 UTC
7 points
0
in reply to: David Matolcsi’s comment on: A relatively brief explanation of Boltzmann Brains
Erogamer “7.!”, (Planecrash=) Project Lawful “the meeting of their minds”, incredibly heavy spoilers on both though much more so Project Lawful.

Eliezer Yudkowsky 18 May 2026 17:08 UTC
14 points
9
in reply to: Lucius Bushnaq’s comment on: A relatively brief explanation of Boltzmann Brains
Well, yes. Any sane choice of UTM is famously pretty equivalent in terms of epistemology, if you’re willing to presume that sanity. Eg, to end up thus convinced against all evidence that Guam is planning to invade the United States, you’d have to twist up the UTM to make it hugely costly to internally simulate other UTMs that would assign greater than negligible probability to “Guam is not planning to invade the United States”. And even then, it washes out in the limit, etcetera.

But if we suppose it to be a prior metaphysical fact out there in the territory, you’d think the territory actually would have to fix some choice of UTM; the territory cannot just say, “Eh, they’re pretty much equivalent” and leave itself uncertain, for all uncertainty exists in the map rather than the territory.

If the territory has picked any sane UTM, all sane UTMs will do well pretty well predicting it.

But how would the territory pick any UTM whatsoever to decide which universes got exactly how much reality-fluid?

Eliezer Yudkowsky 18 May 2026 16:12 UTC
15 points
6
in reply to: JenniferRM’s comment on: A relatively brief explanation of Boltzmann Brains

The whole argument rests on naive/basic logic about complexity being expensive, and yet the actual Solomonoff Prior hasn’t been mentioned?!?

Why should mathematical realism obey the Solomonoff prior? There’s a clear reason from epistemology why the map has to distribute its probability mass more and more sparsely over possible explanations of higher complexity, for in the limit there are countably infinitely many hypotheses of unbounded fine complexity. That’s why, faced with a phenomenon that we think has only one true explanation, we follow Occam’s Razor in our maps.

Why would a Solomonoff prior be out there in the territory? Who is in charge of distributing probability mass over mathematical structures with inhabitants? According to which possible measure of simplicity? What does it mean to pour realityfluid into a mathematical structure?

Mathematical realism answers none of this. To actually try to answer it is to try to come up with a metaphysics of realityfluid. This is beyond anything that Earth can publish in philosophy journals; you would have to look to online erotica for it.

Eliezer Yudkowsky 17 May 2026 20:30 UTC
12 points
8
in reply to: EOC’s comment on: A relatively brief explanation of Boltzmann Brains
One moment of orderly experience that includes remembering an ordered story of how it came to be.

Eliezer Yudkowsky 17 May 2026 19:31 UTC
8 points
2
in reply to: TsviBT’s comment on: A relatively brief explanation of Boltzmann Brains

Instead you could in principle choose to implement a dynamic that concludes “the universe is probably going to soon get bored of being simple, and start being disordered for a while”.

Why don’t you choose to implement that dynamic?

Why, because I don’t expect to see that actually happen to me, of course! In a way that doesn’t depend at all on which anticipation-dynamic I try to implement, even if I could modify my own source code.

Beings that modify their own source code to expect to win the lottery are then promptly surprised by losing the lottery. Beings that modify their preferences to care much more about worlds in which they win the lottery, predominantly see themselves losing the lottery and ending up in the world they cared about less. If you decide to give up on anticipating that your vision stays ordered, it will nonetheless stay quite ordered. The dynamic by which we expect order is not an explanation for that order. “Reality is that which, when you stop believing in it, doesn’t go away.” Even if you were a kind of being that could stop believing in order, it still wouldn’t go away.

Of which I have also said: The art of shepherding is not about using pebbles to control sheep, but letting sheep control pebbles.

Why not unwind our justification loop for expecting order? Why, because at the end of unwinding, we would still see order rather than chaos. You and I both know that’s true. But why is it true? Not the map-why of how do we justify believing it; the territory-why of “How did it end up that way?”