Seems understandable to me (although I guess I’m somewhat primed by reading the previous versions).
npostavs
Karma: 235
I think most of “you” can be omitted in English as well:
Imagine: you study an immature AI in depth. Decode its mind entirely. Develop a great theory of how it works. Validate this theory on a bunch of examples. Use that theory to predict how the AI’s mind will change as it ascends to superintelligence and gains (for the first time) the very real option of grabbing the world for itself. Even then, you are, fundamentally, using a new and untested scientific theory to predict the results of an experiment that has not yet run, about what the AI will do when it really, actually, for real has the opportunity to grab power from the humans.
This seems to be an accidental repost of https://www.lesswrong.com/posts/9TPEjLH7giv7PuHdc/crime-and-punishment-1 from April. (It’s also reposted on https://thezvi.wordpress.com/2025/11/03/crime-and-punishment-1-2/, but not thezvi.substack.com/).
“Von Neumannn was pronounced, by a peer, to be smarter than Albert Einstein to his face and got no objection” interpretation feels off to me
I see that it’s a bit ambiguous, but I read “to his face” as most likely referring to Einstein’s face, which is consistent with your interpretation of Wigner.
The thing that makes hypnosis so bizarre and seemingly powerful is it’s ability to keep attention, [...] [...] In full blown hypnosis [...] they are putting their attention where I specify without doubt or hesitation.
This sounds like it corresponds to “the idea of a state of focused attention”, so I don’t understand why you rejected it. Just because he talks about it as a spectrum (vs a state)? Or something else?
I tried several things without success (each in Claude Opus 4.1, Gemini Pro 2.5, and GPT-5):
Yeah, for now you probably need something more specialized. https://electricalexis.github.io/notagen-demo/ can compose music of semi-decent quality, so with the right training a model ought to be able to manage recognition too (although more unconventional music would be harder).
that I wrote out twice as fast as it actually goes,
Music notation rhythms are relative, so I don’t think this has a real meaning? Like, it might be nicer to use half notes as the main beat, and write the tune mostly in quarters, as you did in the Musescore typeset version. But the hand-written version using eighth notes to a quarter note beat conveys basically the same thing (ignoring the triplet issue).
Your last two Musescore files are missing some separation between 1st and 2nd endings. Compare the images at https://musescore.org/en/handbook/4/voltas
Underdogs lose. If you win, you weren’t the underdog.
Is it not more like, p(underdog_loses) > 0.5? Sometimes the thing with lesser probability happens even if the prediction was well-calibrated.
I don’t think this interpretation can hold up: the body of titotal’s post doesn’t deal with the good vs bad timeline. It’s just about the uncertainty of modelling AI progress which applies for both the good and bad timelines.
I think it’s an intentional pun, like, “whether forecasters” are people who predict whether something will happen or not.
Maybe I should have asked: In what sense are machines “fully doing” first-order logic? I think I understand the part where first logic formulas are recursively enumerable, in theory, but isn’t that intractable to the point of being useless and irrelevant in practice?
Unlike first-order logic, second-order logic is not recursively enumerable—less computationally tractable, more fluid, more human. It operates in a space that, for now, remains beyond the reach of machines still bound to the strict determinism of their logic gates.
In what sense is second-order logic “beyond the reach of machines”? Is it non-deterministic? Or what are you trying to say here? (Maybe some examples would help)
What about tuning the fiddle strings down 1 tone?
You say this:
If you’re thinking, “Wait no, I’m pretty sure my group is fundamentally about X, which is fundamentally good,” then you’re probably still in Red or Blue.
But you also say this:
First, the Grey tribe is about something, [...] things that people already think are good in themselves.
Doesn’t the first statement completely undermine the second one?
I guess you meant jukebox, not jutebox. Unless there is some kind of record-playing box made of jute fiber that I haven’t heard of...
but I recently tried again to see if it could learn at runtime not to lose in the same way multiple times. It couldn’t. I was able to play the same strategy over and over again in the same chat history and win every time.
I wonder if having the losses in the chat history would instead be training/reinforcing it to lose every time.
Yes, my understanding is that the system prompt isn’t really priviledged in any way by the LLM itself, just in the scaffolding around it.
But regardless, this sounds to me less like maintaining or forming a sense of purpose, and more like retrieving information from the context window.
That is, if the LLM has previously seen (through system prompt or first instruction or whatever) “your purpose is to assist the user”, and later sees “what is your purpose?” an answer saying “my purpose is to assist the user” doesn’t seem like evidence of purposefulness. Same if you run the exercise with “flurbles are purple”, and later “what color are flurbles?” with the answer “purple”.
#2: Purposefulness. The Big 3 LLMs typically maintain or can at least form a sense of purpose or intention throughout a conversation with you, such as to assist you.
Isn’t this just because the system prompt is always saying something along the lines of “your purpose is to assist the user”?
by saying their name aloud: [...] …but it’s a lot more difficult to use active recall to remember people’s names.
I’m confused, isn’t saying their name in a sentence an example of active recall?
This is an accidental double post of https://www.lesswrong.com/posts/FJxc4Lk6mijiFiPp2/the-big-nonprofits-post-2025 (also double posted on the wordpress site: https://thezvi.wordpress.com/2025/11/26/the-big-nonprofits-post-2025/ and https://thezvi.wordpress.com/2025/11/27/the-big-nonprofits-post-2025-2/)