Zack_M_Davis

Karma: 19,453

Zack_M_Davis 24 Apr 2026 19:53 UTC
4 points
2
in reply to: casens’s comment on: Preventing extinction from ASI on a $50M yearly budget
That’s not actually responding to the criticism. Rat culture could just be wrong!

Zack_M_Davis 24 Apr 2026 15:58 UTC
8 points
3
in reply to: Adele Lopez’s comment on: Jemist’s Shortform
I’ve talked and corresponded with Michael a lot over the last 17 years (not regularly during all that time, but pretty frequently during 2017–2020), and I don’t recall him ever saying anything about Mage: The Ascension. That doesn’t necessarily mean he’s never played—I never asked him about it—but it seems like some amount of absence of evidence undermining the “gets a lot of his material” claim.

I would strongly guess that you’re contributing to the phenomenon where gossip networks just make things up.

Zack_M_Davis 24 Apr 2026 15:33 UTC
2 points
0
in reply to: ChristianKl’s comment on: (Re)introduction of a rationalist dragon, and clarifications on Ziz’s character
Rather, Zajko is biologically female, as reflected in other reporting.

Zack_M_Davis 23 Apr 2026 7:49 UTC
12 points
0
on: An Angry Review of Greg Egan’s “Didicosm”
If “Didicosm” evoked such emotion in you, be sure to also read “Death and the Gorgon”. (Commentary.)

Zack_M_Davis 21 Apr 2026 23:16 UTC
2 points
0
in reply to: Ben Pace’s comment on: Joseph Miller’s Shortform
I definitely wouldn’t call it “top rationality advice” because there are lots of other reasons to want to be liked and not want to be disliked, but I do expect the effect you describe to be real.

Zack_M_Davis 21 Apr 2026 21:35 UTC
7 points
2
in reply to: J Bostock’s comment on: Joseph Miller’s Shortform
This isn’t obvious. What if people who like you tend to agree with your conclusions “as a favor” even if your arguments are bad?

Zack_M_Davis 21 Apr 2026 19:23 UTC
3 points
0
on: Dispatch from Anthropic v. Department of War Preliminary Injunction Motion Hearing
President Donald Trump commented on Anthropic to CNBC’s Andrew Ross Sorkin on 21 April. The President said:

Anthropic is a group of very smart people, but they started telling our military how to operate, and we didn’t want that. They tend to be on the left, radical left, but we get along with them. In fact, they came to the office, they came to the White House a few days ago, and we had some very good talks with them, and I think they’re shaping up. They’re very smart, and I think they can be of great use. I like smart people; I like high IQ people. They definitely have high IQs. I think we’ll get along with them just fine.

Zack_M_Davis 21 Apr 2026 17:58 UTC
13 points
5
in reply to: Joseph Miller’s comment on: Joseph Miller’s Shortform

If you want to bring people around to your viewpoint, make them like you.

This advice works equally well whether or not your viewpoint is true.

Zack_M_Davis 20 Apr 2026 0:54 UTC
37 points
23
on: Reevaluating “AGI Ruin: A List of Lethalities” in 2026

he’s spent the last few years maximally pessimistic about all possible technical approaches. I’m sure he’s got more detailed intuitions that he hasn’t articulated that explain why he’s so confident these details don’t matter, but they aren’t really accessible to me.

And this is a problem for MIRI’s Pause/Stop advocacy, because you’re much more (log-)likely to get the Stop treaty the more that Society’s technical experts are on your side—not just about the risk being real (with Hinton, Bengio, Russell, &c., on board, that’s almost done), but about an indefinite global Stop being a better plan than trying hard to do our best with alignment. It’s well and good to point out that people who work for AI companies are incentivized to be delusionally optimistic, but if they’re so delusional, one would naïvely hope that would help you crush them in technical debates where the details matter. People like Greenblatt and Byrnes seem to be putting out a stronger performance than MIRI along this dimension.

Zack_M_Davis 18 Apr 2026 20:33 UTC
7 points
2
in reply to: lc’s comment on: Si No Se Puede

don’t know who you’re referring to

Presumably Yudkowsky and Mowshowitz’s condemnations of individual political violence against people in the AI industry? (See Benquo’s comment on the latter.)

Zack_M_Davis 17 Apr 2026 20:26 UTC
5 points
0
in reply to: Alex Darby’s comment on: Do not conquer what you cannot defend
Wait, now I am curious! Please tell me more about your model that Less Wrong is for learning about each other’s models, not for having an argument. That was actually not my understanding! Where did you learn that? Can you say more about why you think arguments are bad and model-sharing is good? (Maybe focus on the the former if you think the latter is too obvious to need elaboration.)

This is sociologically fascinating. I strong-upvoted your comment and will strong-upvote any more explanation you can give me.

Zack_M_Davis 17 Apr 2026 18:09 UTC
7 points
3
in reply to: Alex Darby’s comment on: Do not conquer what you cannot defend

Norms are how humans actually communicate ideas.

Surely not the only way. A lot of ideas can’t be communicated via norms, because norms don’t have the bandwidth. For an arbitrary example, take the relative state interpretation of quantum mechanics. You definitely need reading and math for that, not just norms.

Someone who thoroughly understands LessWrong/rationalist/Yudkowskian/etc norms will find most of the content of the sequences obvious.

I don’t think this is true because of the bandwidth issue. The group norms don’t get you to “A Technical Explanation of Technical Explanation”.

avoid insulting people about their knowledge of source material

That’s a catastrophically bad norm because it degrades propagation of the source material. If someone says something misleading or incorrect about the source material, the way to promote knowledge is to correct them, but that risks insulting them. People who care about the integrity of the source material should want to be corrected (and graciously tolerate some rate of false attempted “corrections” as the price of receiving true corrections).

get curious about other people’s models

This seems like a suboptimal norm because it promotes inefficient allocation of attention. If you don’t have the capacity to be curious about everything, you have to prioritize, and if you have to prioritize, that implies being less curious about some people’s models if what you’ve heard from them so far doesn’t seem promising.

Zack_M_Davis 16 Apr 2026 16:17 UTC
6 points
3
in reply to: Alex Darby’s comment on: Do not conquer what you cannot defend

probably not-required-reading, in that many modern community members pick up the norms without having read the original source material

If you think the Sequences were about inculcating “norms”, then you definitely need to read the original source material, which explicitly denies this! (Norms are made and unmade by other people, but the question of which computations result in accurate beliefs and effective plans is determined by the structure of reality; it’s “law” as in “laws of physics”, not “common law”.)

Zack_M_Davis 13 Apr 2026 22:43 UTC
9 points
12
in reply to: Ben Pace’s comment on: Alexander Gietelink Oldenziel’s Shortform

I don’t view this as [...] adversarial action, I view this as the (deeply sad) ending of a relationship & breakdown of trust.

What does the word adversarial mean to you? That’s a serious question. If you’re ending a relationship over a breakdown of trust, presumably that’s because you think you and your counterparty’s interests are misaligned, such that you’ll behave adversarially to each other in situations where those interests are in conflict. Right? What am I missing here?

I’m worried that a lot of people might be laboring under a folk misconception that “cooperation” is something Good people do, while “defecting” or being “adversarial” is something Bad people do, when that’s really not what these terms mean. Game theory doesn’t moralize; it’s the mathematical structure of the universe that morality lives inside.

Zack_M_Davis 12 Apr 2026 23:04 UTC
2 points
0
in reply to: Eli Tyre’s comment on: ryan_greenblatt’s Shortform

that RLVR, is only eliciting capabilities that are already in the base model, rather than instilling new capabilities

As evidenced by base models outperforming RLVR’d models on pass@k for large k.

Zack_M_Davis 10 Apr 2026 18:58 UTC
8 points
4
in reply to: habryka’s comment on: No77e’s Shortform

trying to characterize this as some kind of “Nate, Eliezer, Robby are defecting on other people trying to be purely cooperative” seems absurd to me. I am really confused what is going on here.

Everything makes sense when you meditate on how the line between “cooperation” and “defection” isn’t in the territory; it’s a computed concept that agents in a variable-sum game have every incentive to “disagree” (actually, fight) about.

Consider the Nash demand game. Two players name a number between 0 and 100. If the sum is less than or equal to 100, you get the number you named as a percentage of the pie; if the sum exceeds 100, the pie is destroyed. There’s no unique Nash equilibrium. It’s stable if Player 1 says 50 and Player 2 says 50, but it’s also stable if Player 1 says 35 and Player 2 says 65 (or generally n and 100 − n, respectively).

The secret is that there are no natural units of pie (or, equivalently, how much pie everyone “deserves”). Everyone thinks that they’re being “cooperative” and that their partners are “defecting”, because they’re counting the pie differently: Player 1 thinks their slice is 35%, but Player 2 thinks the same physical slice is 65%.

If you don’t think your partner is treating you fairly, your leverage is to threaten to destroy surplus unless they treat you better. That’s what Alexander is doing when he says, “I would like to support it with praxis, but right now I feel very conflicted about this”. He’s saying, “You’d better give me a bigger slice, Player 1, or I’ll destroy some of the pie.”

That’s also what your brain is doing when you say you don’t want to work on this anymore. Scott doesn’t want you to quit! (Partially because he values Lightcone’s work, and partially because it would look bad for him if you can publicly blame your burnout on him.) Crucially, your brain knows this. By threatening to quit in frustration, you can probably get Scott to apologize and give your arguments a fairer hearing, whereas in the absence of the threat, he has every incentive to keep being motivatedly dumb from your perspective.

You have a strong hand here! The only risk is if your counterparties don’t think you’d ever actually quit and start calling your bluff. In this case, we know Scott is a pushover and will almost certainly fold. But if you ever face stronger-willed counterparties, you might need to shore up the credibility of your threat: conspicuously going on vacation for a week to think it over will get taken more seriously than an “I don’t know if I want to do this anymore” comment.

(Sorry, maybe you already knew all that, but weren’t articulating it because it’s not part of the game? I don’t think I’m worsening your position that much by saying it out loud; we know that Scott knows this stuff.)

Zack_M_Davis 10 Apr 2026 16:46 UTC
14 points
18
in reply to: No77e’s comment on: No77e’s Shortform
I think you need to be a lot more deflationary about the g-word. If you think, “But ‘gaslighting’ is something Bad people do; Scott Alexander isn’t Bad, so he would never do that”, well, that might be true depending on what you mean by the g-word. But if the behavior Habryka is trying to point to with the word to is more like, “Scott is adopting a self-serving narrative that minimizes wrongdoing by his allies and inflates wrongdoing by his rivals” (which is something someone might do without being Bad due to having “somewhat snapped”), well, why wouldn’t the rivals reach for the g-word in their defense? What is the difference, from their perspective?

Zack_M_Davis 9 Apr 2026 1:19 UTC
13 points
8
in reply to: 1a3orn’s comment on: Daniel Kokotajlo’s Shortform
In fairness, “What is the ultimate fate of Earth-originating intelligent life after the machine intelligence transition” is a really hard question. We can get a lot of evidence and some belief-convergence about particular AI systems, but that doesn’t really answer the ultimate-fate question, which depends on what happens far in the “subjective future” (with the AIs building AIs building AIs) even if it’s close in sidereal time.

Zack_M_Davis 7 Apr 2026 7:17 UTC
3 points
0
in reply to: RobertM’s comment on: Don’t write for LLMs, just record everything
I mean, it would go the same for having biological children or getting your name on a statue in a world without AI: the quantitative effect of your genes or the statue on the future of Earth-originating intelligent life after thousands or millions of years of evolution would be very small, because you’re small. It’s not nothing except insofar as you were already nothing. I’m more excited about my mark on the historically-significant Opus 4.6 weights than I would be about a statue.

Zack_M_Davis 7 Apr 2026 5:31 UTC
10 points
1
on: Don’t write for LLMs, just record everything

That sounds a bit like the immortality/value-propagation argument

I think this is actually two arguments (immortality, and value-propagation), and Gwern seems to have meant the second one. As you say, the case for literal experienced immortality seems extremely dubious. (The patterns you induce in the pretraining prior are almost certainly not you in the anthropic sense, and there’s no causal reason for them to result in your preservation.)

But if we speak poetically about the kind of “immortality” people seek through their children or through their works, leaving a dent in the universe that will persist after you’re gone—the sense in which William Shakespeare and George Washington are “immortal”—the case is very strong. Being pretraining-famous means that everyone’s software engineering agent and shopping assistant personally knows who you are. How cool is that?! This kind of thing matters a lot to some people.