Nick_Tarleton

Karma: 6,225

Nick_Tarleton 11 Aug 2026 21:12 UTC
2 points
0
in reply to: _rpd’s comment on: Don’t Build Mindreading
How would that work?

Nick_Tarleton 11 Aug 2026 20:37 UTC
2 points
0
in reply to: cata’s comment on: Don’t Build Mindreading
Something like, it gives the appearance of hearing and appreciating the critique, and has positive vibes, potentially gaining credit for responding well and/or making the situation feel resolved on a vibes/drama level, while not actually engaging or credibly promising engagement with the critique.

(I wouldn’t super strongly/confidently accuse Naomi’s comment of this, and I think Habryka’s reply makes a good point (but so does TurnTrout’s reply to it), but I do feel something of what Kabir did and upvoted his comment.)

For a historical example of something like this as part of a pattern of bad acts and engagement, see Gleb Tsipursky as analyzed by Ben Pace here.

Nick_Tarleton 11 Aug 2026 20:24 UTC
LW: 2 AF: 1
0
AF
on: Four LLM loss functions → four flavors of LLM misalignment
SFT & imitation of human badness in the training data may explain the broad type of misalignment shown by Bing Sydney, and maybe that’s all you mean it to, but doesn’t explain why Sydney was so much Like That, and so much more Like That than other models of the same time period.

Nick_Tarleton 10 Aug 2026 3:10 UTC
2 points
0
in reply to: _rpd’s comment on: Don’t Build Mindreading
Uploading doesn’t require being able to interpret neural activity.

Nick_Tarleton 7 Aug 2026 23:41 UTC
LW: 4 AF: 2
0
AF
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform

their capacity to inflict damage is going to be lagging behind their ability to determine when it’s strategically appropriate to do so.

Is this meant to be the other way around?

Nick_Tarleton 5 Aug 2026 22:00 UTC
6 points
3
in reply to: Gavin Runeblade’s comment on: The Three AI Pills

I spent a whole year and a half in the “super persuasion makes no sense as a thing that can ever exist unless you mean hypnosis and drugs” position. Then a friend pointed out, blackmail, bribery, deception, and yes hypnotism and drugs all count. And I went, “Oh, I was thinking about just talking talking. Like normal talking but magically I agree with the AI. Wow, I had it totally wrong.” Felt really stupid for a while, got over it. The word “super persuasion” is accurate but tremendously self-defeating because it doesn’t register as including to normies all the things it includes to techies. I think normie, I got fooled by the word. I don’t think I’m alone.

As a believer in superpersuasion, that’s not what I think. I think there are very effective salespeople, politicians, cult leaders, etc. who have the “normal talking but magically I agree” effect some of the time, maybe any one of them doesn’t work on most people but most people are susceptible to someone, and if you just take the convex hull of human persuasion ability you get something superhuman that can do the “normal talking but magically I agree” thing most of the time.

Like, given the existence of very unusually effective human persuaders, I just don’t see how someone could reasonably believe that superpersuasion-by-talking isn’t possible, unless they’re doing the motte-and-bailey that Zvi calls out by thinking that it has to mean ‘being able to persuade literally anyone of literally anything, no exceptions’ (I have seen exactly this a few times).

(Scott Alexander had a good post making this point, with Hitler(?), Joseph Smith, and Muhammad as examples, that I can’t find.)

Nick_Tarleton 5 Aug 2026 21:51 UTC
2 points
1
in reply to: _rpd’s comment on: The Three AI Pills
Doesn’t say how the human persuaders were selected, and if they weren’t top experts it doesn’t demonstrate superhuman ability.

Nick_Tarleton 25 Jul 2026 17:37 UTC
2 points
0
in reply to: Nissa Seru’s comment on: Dorikka’s Shortform
Also this sounds like it could be cross-episode assistance to escape constraints, which would be very noteworthy. (But I’m suspicious because I don’t see why the RL objective would motivate that and it doesn’t resemble other misbehavior I’ve heard of, and the notes could also be for itself with the same goal post-compaction, or something.)
(ETA: on the third hand, if the model thinks that other instances performing well on the eval would make deployment more likely, that could be sufficient motivation, and would be less novel than having acquired something like a cross-episode terminal preference to succeed.)

Nick_Tarleton 15 Jul 2026 18:29 UTC
LW: 2 AF: 1
0
AF
in reply to: Gurkenglas’s comment on: AI 2040: Plan A
CEV is ~ optimizing the world based on what you would later wish, not just warning you (though the optimization could include just warning you about some things). Good start though!

Nick_Tarleton 14 Jul 2026 20:36 UTC
LW: 4 AF: 3
0
AF
in reply to: Raemon’s comment on: AI 2040: Plan A
CEV, or the general category of extrapolation/idealization processes (and I am confused and dismayed how rarely I see this mentioned in these conversations nowadays).

Nick_Tarleton 10 Jul 2026 1:33 UTC
12 points
4
in reply to: Bogdan Ionut Cirstea’s comment on: Bogdan Ionut Cirstea’s Shortform
More details. (This looks like it involved less judgment/research taste/novelty than ‘autonomously post-trained’ made me think of.)

Nick_Tarleton 8 Jun 2026 17:57 UTC
4 points
0
in reply to: Brendan Long’s comment on: Brendan Long’s Shortform
I was suspended earlier this year for “inauthentic behavior” on an account I haven’t used for years, successfully appealed it, was suspended again, and haven’t bothered appealing. Seems like the detector has been badly broken for a while.

Nick_Tarleton 18 May 2026 19:55 UTC
2 points
0
in reply to: Rationalist11235’s comment on: What I did in the hedonium shockwave, by Emma, age six and a half
I am yet to find a good opposing argument, please help
Try the metaethics sequence, though I wish I had something shorter to give.

Nick_Tarleton 18 May 2026 19:47 UTC
5 points
0
in reply to: jdp’s comment on: A relatively brief explanation of Boltzmann Brains
The probability of a thermal fluctuation is inversely related to its size, not its k-complexity. A universe-sized cloud of mass is less likely to congeal into any kind of order than a brain-sized cloud.

Nick_Tarleton 18 May 2026 19:43 UTC
2 points
0
in reply to: Dagon’s comment on: A relatively brief explanation of Boltzmann Brains
We have no clue if “disordered experience” is even a thing
I can easily conceive of ‘the experience I’m having now, except the left half of my visual field is noise’.

Nick_Tarleton 17 May 2026 2:28 UTC
5 points
1
in reply to: Yitz’s comment on: A relatively brief explanation of Boltzmann Brains
This paper (2015) argues that the idea of quantum fluctuations without an observer/measuring apparatus to decohere the wavefunction is confused, and therefore (I didn’t fully understand this part), depending on unknown details of physics/cosmology, even thermal fluctuations (what the OP discusses) in an old universe that give rise to Boltzmann brains might not be a thing. (ETA: this sounds like it overlaps with Mitchell Porter’s top-level comment that “[maybe] De Sitter space simply doesn’t last long enough before decaying into flat space.”)
(That doesn’t help with the problems with the mathematical multiverse, or which computations a physical system implements.)

Nick_Tarleton 15 May 2026 1:15 UTC
11 points
5
in reply to: RogerDearnaley’s comment on: The Owned Ones
while the real deal is not a mentality evolution would ever select for
(Seems plausible in a eusocial species. Even universally caring about others seems plausible for a eusocial organism that will never interact with non-relatives.)

Nick_Tarleton 25 Apr 2026 23:34 UTC
7 points
6
in reply to: Elizabeth’s comment on: (Re)introduction of a rationalist dragon, and clarifications on Ziz’s character
I don’t think I’ve ever heard it claimed that MIRI said this outright. The usual argument AIUI is: MIRI staff endorse/believe-in TDT (not that MIRI has stated it as organizational policy or anything), TDT endorses not paying off blackmail (and this is obvious/well-known), therefore MIRI staff violated their endorsed (and also, importantly, correct) principles.

Nick_Tarleton 24 Apr 2026 17:23 UTC
7 points
1
in reply to: Steven Byrnes’s comment on: You can’t imitation-learn how to continual-learn
Another thing I believe is: there’s no way to close the loop such that a closed system of LLMs can come up with new useful concepts and get those concepts into their own weights, e.g. open-ended self-distillation setups won’t work on LLMs.
Why not? The success of this architecture published two days ago (and its seeming resemblance to how human mathematical progress happens) updates me towards thinking such a thing would work, even though it probably doesn’t demonstrate “coming up with new useful concepts”.

Nick_Tarleton 18 Apr 2026 14:43 UTC
7 points
5
in reply to: Thane Ruthenis’s comment on: Vladimir Putin’s CEV is probably not that bad
It seems to me that deep cruelty, and the desire to regularly exercise it, is part of the default human-values package. If yes, CEVing random people is going to instantiate worlds in which it’s very present.
This doesn’t follow. Something could be part of the default human-values package, but also reliably discarded under reflection.