gustaf

Karma: 78

gustaf 10 Nov 2025 22:10 UTC
13 points
0
on: From Vitalik: Galaxy brain resistance
Crossposting Eliezer’s comments.
Thread between Eliezer and Vitalik
Eliezer Yudkowsky replying on Twitter:
This applies way beyond mere ethics, though! As a kid I trained myself by trying to rationalize ridiculous factual propositions, and then for whatever argument style or thought process reached the false conclusion, I learned to myself: “Don’t think that way.”
Vitalik Buterin:
Indeed, but I think ethics (in a broad sense) is the domain where the selection pressure to make really powerful galaxy brain arguments is the strongest.
Outside of ethics, perhaps self-control failures? eg. the various “[substance] is actually good for me” stories you often hear. Though you can model these as being analogous, they’re just about one sub-agent in your mind trying to trick the others (as opposed to one person trying to trick other people).
Eliezer Yudkowsky:
Harder training domain, not so much because you’re more tempted to fool yourself, as because it’s not clear-cut which propositions are false. I’d tell a kid to start by training on facts and make sure they’re good at that before they try training on ethics.
Vitalik Buterin: Agree!
Eliezer Yudkowsky:
Eg:
Argue: That a saltshaker is Jesus Christ.
Rationalization: Gurer jrer n ybg bs ngbzf va Wrfhf’f obql naq fbzr bs gurz ner ab qbhog va guvf fnygfunxre, tvira Nibtnqeb’f ahzore naq gur ibyhzr bs Rnegu’f fhesnpr.
Lesson: Look at what your thoughts did there; never do that.
Eliezer Yudkowsky used ROT13 to hide an example Rationalization.
After thinking about it yourself, hover over the box below, to reveal his example.
There were a lot of atoms in Jesus’s body and some of them are no doubt in this saltshaker, given Avogadro’s number and the volume of Earth’s surface.
Related writing by Eliezer
Skill: Anti-rationalization (1): Prevent your mind from selectively searching for support of only one side of an argument.
Subskill: Notice the process of selectively searching for support, and halt it. Chains into “Ask whether, not why”, or into trying to unbind the need to rationalize. Level 1, notice the active process after performing a conscious check whether it’s running; Level 2, detect it perceptually and automatically; Level 3, never run this process.
Exercise material: List of suggested conclusions to rationalize. I’m not quite sure what the prerequisites here would be; one of my own childhood epiphanies was finding that I could argue reasons why there ought to be a teapot in the asteroid belt, and thinking to myself, “Well, I better not do that, then.” I suspect the primary desideratum would be more that the topic offers plenty of opportunity to come up with clever rationalizations, rather than that the topic offers motivation to come up with clever rationalizations.
Exercise A: Pick a conclusion from the list. Come up with a clever argument why it is true. Notice what it feels like to do this. Then don’t do it.
(This is distinct from noticing the feeling of being tempted to rationalize, a separate subskill.)
Exercise B: In the middle of coming up with clever arguments why something is true, stop and chain into the Litany of Tarski, or some other remedy. Variant: Say out loud “Ew!” or “Oops!” during the stop part.
Exercise C: For a conclusion on the list, argue that it is true. Then “Ask whether, not why”—try to figure out whether it is true. Notice the difference between these two processes. Requires a question whose answer is less than immediately obvious, but which someone can find info about by searching their memory and/or the Internet.
https://www.lesswrong.com/posts/f4CZNEHirweN3XEjs/teachable-rationality-skills?commentId=F837zHgkrTA2rqusn
I learned at a very young age not to be impressed by the sleight-of-hand that assembles a clever-sounding argument for why there might be a teapot in the asteroid belt, even or especially when it was my own brain generating the argument; that was where my journey began, with the search for a kind of thinking that couldn’t argue impressively for any conclusion.
https://www.reddit.com/r/rational/comments/dk4hh8/comment/f4po393
When I was a kid I trained myself by coming up with the best arguments I could for propositions I knew to be false, then contemplating the arguments until they no longer seemed persuasive. I picked ridiculous targets.
https://x.com/ESYudkowsky/status/1900621719225987188
See Also: Against Devil’s Advocacy

gustaf 10 Nov 2025 14:27 UTC
2 points
0
in reply to: Saul Munn’s comment on: Saul Munn’s Shortform
https://johnvon.com (clickable link)
Lukas Münzel (who made this project):
I got access to over a dozen unpublished documents by John von Neumann.
[...]
Some particularly interesting passages:
1. Von Neumann on reviewing a student’s work—“I have made an honest effort to formulate my objections as little brutally as possible” [Original / German / English]
2. A young von Neumann very politely asking for a warm intro to Schrödinger, then 39 years old [Original / German / English]
3. Von Neumann presenting a very significant result, the “Von Neumann’s mean ergodic theorem”, to his mentor. Fascinatingly, he also expresses concern in this 1931 letter from princeton that “[the USA] looks roughly like Germany in 1927″ [Original / German / English]
See also: Methodology and Code for the transcription/translation via GPT-5-Thinking and Claude-Sonnet-4.5.^[1]
They likely accessed these via ETH Zurich’s library.^[2]
1. They used gpt-5 (line 14 & line 83) and gpt-5 in API = gpt-5-thinking in system card. Claude in line 46. ↩︎
2. Lukas studies at ETH Zurich. HS 91:682 is in ETH Zurich’s library ↩︎

gustaf 29 Oct 2025 10:20 UTC
3 points
0
on: Feature request: draft comments
For future readers: this feature has now been shipped.

gustaf 27 Oct 2025 10:17 UTC
5 points
0
on: Credit goes to the presenter, not the inventor
For background context see EDVAC—Wikipedia

See also:
- Stigler’s law of eponymy: “No scientific discovery is named after its original discoverer”
- Boyer’s law: “Mathematical formulas and theorems are usually not named after their original discoverers”
- Matthew effect / “The rich get richer and the poor get poorer”
- @roon on X: “the person who names a thing is often more powerful than the real discoverer”
- Beware Amateur Science History—Robin Hanson
- Berry’s 3 laws of discovery
  - Referencing Arnold, who also referenced Berry in “On teaching mathematics”
- More eponymous laws

gustaf 17 Oct 2025 16:45 UTC
3 points
0
on: AI #138 Part 1: The People Demand Erotic Sycophants
NotebookLM now works directly with arXiv papers. I don’t want their podcasts, but if they get Gemini 3.0 plus easy chat with an arXiv paper and related materials, cool.

Your link is not about Google’s NotebookLM. This is a NotebookLM-like feature from alphaXiv. “easy chat with an arXiv paper” in alphaXiv has been support since 2025-05-10. NotebookLM has already supported arXiv for a while: you can paste the arXiv PDF link into NotebookLM.

minor errors

Aaron Silverbook got a $5k ACX grant

This should link to https://www.astralcodexten.com/p/acx-grants-results-2025#:~:text=do the same.-,Aaron Silverbook%2C %245K,-%2C for approximately five instead of https://x.com/AiDigest_/status/1977781138442916158

Also some typos:
- “Sonet” → “Sonnet”
- “Schilling point” → “Schelling Point”
- “InterfaceMax” → “InferenceMax”

gustaf 9 Oct 2025 17:19 UTC
4 points
0
in reply to: daijin’s comment on: daijin’s Shortform
Maybe Slightly Skew Systems Of Government

gustaf 7 Oct 2025 19:26 UTC
1 point
0
on: LessWrong Reacts
This wikitag should be optimized for:
1. Illustrating this feature. A reader unfamiliar with the feature should go like “ah, yeah that seems useful” or even “ooh, LessWrong is awesome!”
2. Explain confusing reacts to those not already in the know
I originally arranged confusing-first to optimize 2., but Ben Pace’s comment reminded me of 1.

See also: Ben Pace introducing new reacts + people’s replies.

gustaf 7 Oct 2025 19:20 UTC
1 point
0
in reply to: Ben Pace’s comment on: Ben Pace’s Shortform Feed
Having some common ones first makes sense for illustrating the feature! I arranged them based on which reacts would probably be the most confusing.

gustaf 7 Oct 2025 18:09 UTC
3 points
0
in reply to: Ben Pace’s comment on: Ben Pace’s Shortform Feed
Thanks for the wiki page stub. I turned it into a full table (See LessWrong Reacts)

I think this presents a nice opportunity of explaining LessWrong’s culture to new users.

Have you considered linking to the wiki page from the hover-menu of a reaction?

I think the most confusing one, even after hovering over the reaction is “I beseech you”

gustaf 7 Oct 2025 16:55 UTC
5 points
0
in reply to: RobertM’s comment on: RobertM’s Shortform
Small bug that might confuse users: the new draft shows up where I wrote it, only after I reload the page.

Repro:
1. On post page write comment replying to another comment
2. “Save As Draft”
3. Observation: draft does not show up under the parent comment
4. Reload page
5. Observation: draft does show up under the parent comment
Expectation: at 3., the draft should appear under the parent comment

If I had to hazard a guess, I would suspect what the codebase calls “a terrible hack” is at fault. If I were to venture further, I would guess getDraftSelector implies the default is include-my-draft-replies, while line 343 of the terrible hack implicitly assumes drafts are excluded by default.

gustaf 7 Oct 2025 15:39 UTC
1 point
0
on: LessWrong Reacts
I imported the reactions from the LessWrong source code into a table, that users can now add explanations to. I did this programmatically, which took me a few tries to get the formatting correct. (For example getting the correct image sizes, not just on the page, but in the editor.) I eventually generated HTML and updated with type ckEditorMarkup using the API.

gustaf 7 Oct 2025 13:14 UTC
1 point
0
in reply to: quetzal_rainbow’s comment on: “Intelligence” → “Relentless, Creative Resourcefulness”
It seems you recall correctly. The old definition seemed to additionally include “efficient”.

Muehlhauser & Salamon (2012):

For our purposes, “intelligence” measures an agent’s capacity for efficient cross-domain optimization of the world according to the agent’s preferences.

Yudkowsky (2013) (also expands on this definition):

I usually use a notion of “intelligence ≡ efficient cross-domain optimization,”

See also Efficient Cross-Domain Optimization — Eliezer Yudkowsky

gustaf 7 Oct 2025 12:44 UTC
1 point
0
in reply to: Raemon’s comment on: niplav’s Shortform
For future readers: Raemon expanded on this in a top-level post at “Intelligence” → “Relentless, Creative Resourcefulness”

gustaf 4 Oct 2025 21:24 UTC
3 points
0
in reply to: mrpalmtree19’s comment on: mrpalmtree19′s Shortform
Here are some things I did:
1. tell my friends and family about the book
2. write a review on Amazon & Goodreads
3. review scam copies on Amazon with 1 star
  - Searching for IABED on amazon.com, you will find that after the real book, several LLM written scam copies show up.
  - the same is true for amazon.co.uk, .de, … (these are distinct as far as reviews go)
  - I looked at the sample text and if it made no sense and was LLM written (it always was), wrote a 1 star review (only on amazon.de)
  - I only rated on amazon.de, because my Amazon account is only eligible for that. You could do it for the other Amazon sites.
  - Usually my reviews are display on the next day
  - (MIRI is aware but “it seems like there isn’t that much we can do about it, unfortunately.”)
  - (I also tried reporting them; but I am not sure they are breaking any rules.)
4. request the book in libraries
  - I found a list of German public libraries, where one can easily request them to buy a book and went through that. You could do this for libraries in your region.
  - I did 15 requests. 4 libraries bought it. 2 said they won’t buy it.
I’d attend a march, if 1000 people also pledged to march in Germany.

I might contact my representatives (How to email your politician).

gustaf 4 Oct 2025 21:15 UTC
14 points
0
on: gustaf’s Shortform
Here are some things I did after reading If Anyone Builds It, Everyone Dies:
1. tell my friends and family about the book
2. write a review on Amazon & Goodreads
3. review scam copies on Amazon with 1 star
  - Searching for IABED on amazon.com, you will find that after the real book, several LLM written scam copies show up.
  - the same is true for amazon.co.uk, .de, … (these are distinct as far as reviews go)
  - I looked at the sample text and if it made no sense and was LLM written (it always was), wrote a 1 star review (only on amazon.de)
  - I only rated on amazon.de, because my Amazon account is only eligible for that. You could do it for the other Amazon sites.
  - Usually my reviews are display on the next day
  - (MIRI is aware but “it seems like there isn’t that much we can do about it, unfortunately.”)
  - (I also tried reporting them; but I am not sure they are breaking any rules.)
4. request the book in libraries
  - I found a list of German public libraries, where one can easily request them to buy a book and went through that. You could do this for libraries in your region.
  - I did 15 requests. 4 libraries bought it. 2 said they won’t buy it.
I’d attend a march, if 1000 people also pledged to march in Germany.

I might contact my representatives (How to email your politician).

gustaf 3 Oct 2025 16:12 UTC
1 point
0
on: The Grand Encyclopedia of Eponymous Laws
Here is another one:

Yudkowsky’s Law of Ultrafinite Recursion states that “in practice, infinite recursions are at most three levels deep.”

gustaf 18 Jul 2025 14:24 UTC
17 points
0
in reply to: Richard_Kennaway’s comment on: Comment on “Four Layers of Intellectual Conversation”
With LessWrong, HPMoR and Planecrash you got the bulk covered. You could skim the following overview of Eliezer Yudkowsky’s writing:
- on LessWrong starting with The Sequences / Rationality A-Z (book)
- Harry Potter and the Methods Of Rationality (book)
- Planecrash / Project Lawful (long fictional story)
- on Arbital (AI Alignment)
- on Twitter/X (mostly retweeting)
- on Facebook (mostly retweeting)
- on Medium (2 posts)
- on Tumblr (fiction / on writing)
  - e.g. Masculine Mongoose (short story)
- on fanfiction.net (3 stories)
- on yudkowsky.net (personal/fiction)
- on Reddit (fiction / on writing)
  - also pseudonymously: Kindness to Kin

gustaf’s Shortform

gustaf18 Jul 2025 14:18 UTC

2 points

2 comments1 min readLW link

gustaf 18 Jul 2025 14:18 UTC
1 point
0
on: gustaf’s Shortform
Overview of Eliezer Yudkowsky’s writing:
- on LessWrong starting with The Sequences / Rationality A-Z (book)
- Harry Potter and the Methods Of Rationality (book)
- Planecrash / Project Lawful (long fictional story)
- on Arbital (AI Alignment)
- on Twitter/X (mostly retweeting)
- on Facebook (mostly retweeting)
- on Medium (2 posts)
- on Tumblr (fiction / on writing)
  - e.g. Masculine Mongoose (short story)
- on fanfiction.net (3 stories)
- on yudkowsky.net (personal/fiction)
- on Reddit (fiction / on writing)
  - also pseudonymously: Kindness to Kin

gustaf 4 Apr 2025 12:22 UTC
1 point
0
on: Exercise: Planmaking, Surprise Anticipation, and “Baba is You”
Just had the idea to share my experience when I tried this in February. Previous Experience: Had played world 1 before Goal: Actually get it right on the first try with ~90% confidence

Approach:
- I wanted high legibility on my thoughts, to increase surface area, that I can reflect on
- thinking out loud with screen + audio recording
- thinking through writing (e.g. “assumption: …”, “plan A”, “realization”, “insight”)
- durations (e.g. 4min) to allow me to see my speed at a glance
- Fatebook predictions for my solution working
- Feb08 at 20:00 I played world 2 level 1: took 45min + 20min of reflection
- Feb23 at 05:40 I played world 2 level 2: took 40min + 20min of reflection
- Feb23 at 07:00 I played world 2 level 3: was quite easy, so I found 5 solution candidates. My confidence ranging from 65%-90%; all ended up working
My initial idea for both level 1 and level 2 would have failed though I knew to continue and then found a correct solution. I like that this exercise gave me a glimpse of what it feels like to solve a problem that you felt stuck on. I think it is useful for me continue this exercise

If it’s useful feedback to you, I can send you my notes (3 levels) and the recording (only first level)

gustaf

Thread between Eliezer and Vitalik

Related writing by Eliezer

minor errors

gustaf’s Shortform