davekasten

Karma: 1,595

davekasten 20 Oct 2025 5:45 UTC
2 points
0
in reply to: Zach Stein-Perlman’s comment on: davekasten’s Shortform
To be clear, I think people should feel free to block freely and for any reason, including literally no reason at all. I’m open to ways of describing people’s block decisions in the future that better convey that, but I definitely didn’t think others reading this would assume “oh, Zach’s the bad guy here” as opposed to the reverse.

davekasten 20 Oct 2025 5:38 UTC
2 points
0
in reply to: habryka’s comment on: davekasten’s Shortform
By the “whitepaper,” are you referring to the RSP v2.2 that Zach linked to, or something else? If so, I don’t understand how a generic standard can “call out what kind of changes would need to be required” to their current environment if they’re also claiming they meet their current standard.
Also, just to cut a little more to brass tacks here, can you describe the specific threat model that you think they are insufficiently responding to? By that, I don’t mean just the threat actor (insiders within their compute provider) and their objective to get weights, but rather the specific class or classes of attacks that you expect to occur, and why you believe that existing technical security + compensating controls are insufficient given Anthropic’s existing standards.
For example, AIUI the weights aren’t just sitting naively decrypted at inference, they’re running inside a fairly locked down trusted execution environment, with keys provided only as-needed (probably with an ephemeral keying structure?) from a HSM, and those trusted execution environments are operating inside a physical security perimeter of a data center that already is designed to mitigate insider risk. Which parts of this are you worried are attackable? To what degree are organizational boundaries between Anthropic and its compute providers salient to increasing this risk? Why should we expect that the compute providers don’t already have sufficient compensatory controls here, given that, e.g., these compute providers also provide classified compute to the US government that is secured at the Top Secret / SCI level and presumably therefore have best-in-class anti-insider-threat capabilities?
I’m extremely willing to buy into a claim that they’re not doing enough, but I would actually need to have an argument here that’s more specific.

davekasten 19 Oct 2025 23:27 UTC
2 points
0
in reply to: habryka’s comment on: davekasten’s Shortform
I’m really not following your argument here. Of course in many instances compute providers don’t offer zero trust relationships with those running on their systems. This is just not news. There’s a reason why we have an entire universe of compensating technical and non-technical controls to mitigate risk in such circumstances.
You have done zero analysis to identify any reason to believe that those compensating controls are insufficient. You could incredibly easily get me to flip sides in this discussion if you offered any of that, but simply saying that someone’s running zero trust isn’t sufficient. As a hypothetical, if Anthropic is expending meaningful effort to be highly confident that they can ensure that Amazon’s own security processes are securing against insiders, they would have substantial risk reduction (as long as they can have high confidence said processes are continuing to be executed.
Separately, though it probably cuts against my argument above [1], I would politely disagree with the perhaps-unintended implication in your comment above that “implement zero trust” is a sufficient definition of defenses to defend against compute providers like Amazon, MSFT, etc. After all, Anthropic’s proper threat modeling of them should include things like, “Amazon, Microsoft, etc. employ former nation-state hackers who considered attacking zero trust networks to be part of the cost of doing business.”
[1] Scout mindset, etc.

davekasten 17 Oct 2025 3:52 UTC
2 points
−4
in reply to: MichaelDickens’s comment on: davekasten’s Shortform
Huh? Simply using someone else’s hosting doesn’t mean that Amazon has a threat-modeled ability to steal Claude’s model weights.

For example, it could be the case (not saying it is, this is just illustrative) that Amazon has given Anthropic sufficient surveillance capabilities inside their data centers that combined with other controls the risk is low.

davekasten 17 Oct 2025 3:50 UTC
2 points
0
in reply to: habryka’s comment on: davekasten’s Shortform
Where’s the “almost certainly” coming from? I feel like everyone responding to this is seeing something I’m not seeing.

davekasten 11 Oct 2025 18:28 UTC
11 points
0
on: davekasten’s Shortform
Zach Stein-Perlman’s recent quick take is confusing. It just seems like an assertion, followed by condemnation of Anthropic conditioned on us accepting his assertion blindly as true.

It is definitely the case that “insider threat from a compute provider” is a key part of Anthropic’s threat model! They routinely talk about it in formal and informal settings! So what precisely is his threat model here that he thinks they’re not defending adequately against?

(He has me blocked from commenting on his posts for some reason, which is absolutely his right, but insofar as he hasn’t blocked me from seeing his posts, I wanted to explicitly register in public my objection to this sort of low-quality argument.)

davekasten 30 Sep 2025 18:11 UTC
4 points
0
in reply to: Buck’s comment on: Buck’s Shortform
My opinion, FWIW, is that both treaty and international agreement (or “deal”, etc.) have upsides and downsides. And it’s hard to predict those considerations’ political salience or direction in the long term—e.g., just a few years ago, Republicans’ main complaint against the JCPOA (aka “the Iran Nuclear Deal”) was that it wasn’t an actual treaty, and should have been, which would be a very odd argument in 2025.

I think as long as MIRI says things like “or other international agreement or set of customary norms” on occasion it should be fine. It certainly doesn’t nails on the chalkboard me to hear “treaty” on a first glance, and in any long convo I model MIRI as saying something like “or look, we’d be open to other things that get this done too, we think a treaty is preferable but are open to something else that solves the same problem.”

davekasten 24 Sep 2025 22:39 UTC
5 points
0
in reply to: MalcolmMcLeod’s comment on: The title is reasonable
The big challenge here is getting national security officials to respond to your survey! Probably easier with former officials, but unclear how much that’s predictive of current officials’ beliefs.

davekasten 22 Sep 2025 4:24 UTC
8 points
7
on: The title is reasonable
I’m pretty sure that p(doom) is much more load-bearing for this community than policymakers generally. And frankly, I’m like this close to commissioning a poll of US national security officials where we straight up ask “at percent X of total human extinction would you support measures A, B, C, D, etc.”

I strongly, strongly, strongly suspect based on general DC pattern recognition that if the US government genuinely belived that the AI companies had a 25% chance of killing us all, FBI agents would rain out of the sky like a hot summer thunderstorm, sudden, brilliant, and devastating.

davekasten 20 Sep 2025 17:20 UTC
175 points
43
on: davekasten’s Shortform
Heads up—if you’re 1. on a H1-B visa AND 2. currently outside the US, there is VERY IMPORTANT, EXTREMELY TIME SENSITIVE stuff going on that might prevent you from getting back into the US after 21 September.

If this applies to you, immediately stop looking at LessWrong and look at the latest news. (I’m not providing a summary of it here because there are conflicting stories about who it will apply to and it’s evolving hour by hour and I don’t want this post to be out of date)

davekasten 18 Sep 2025 7:05 UTC
9 points
1
in reply to: MalcolmMcLeod’s comment on: How To Dress To Improve Your Epistemics
Ivy Style Any% Speedrun Complete

davekasten 15 Sep 2025 20:29 UTC
3 points
0
on: davekasten’s Shortform
If you’re someone who has^[1], or will have, read If Anyone Builds It, Everyone Dies, I encourage you to post your sincere and honest review of the book on Amazon once you have read it—I think it would be useful to the book’s overall reputation.
But be a rationalist! Give your honest opinion.
When:
If you’ve already read it: Once Amazon accepts reviews, likely starting on the book launch date tomorrow.
If you haven’t read it: Once you’ve read it. Especially if you’ve ordered a copy from Amazon so they know the review is coming from a verified purchaser of the book.
1. ^
  Advance reader copies.

davekasten 9 Sep 2025 2:27 UTC
47 points
18
in reply to: chanamessinger’s comment on: chanamessinger’s Shortform
I also think this is likely to cause folks to look into the situation and ask, “is it really this bad?” I think it’s helpful to point them to the fact that yes, Yudkowsky and Soares are accurately reporting that the AI CEOs think they’re roughly russian-roulette odds gambling with the world [1]. I also think it’s important to emphasize that a bunch of us have a bunch of disagreements, whether nuanced or blunt, with them, and still are worried.

Why? Because lots of folks live in denial that it’s even possible for AI as smart as humans to exist one day, much less superintelligent AI soon. Often their defense mechanism is to pick at bits of the story. Reinforcing that even if you pick at bits of the story you still are worried is a helpful thing.
[1] Not trying to pick round ninety zillion of the fight about whether this is a good or bad idea, etc.!

davekasten 5 Sep 2025 5:38 UTC
5 points
2
in reply to: ryan_greenblatt’s comment on: Anthropic’s leading researchers acted as moderate accelerationists
I honestly haven’t thought especially in depth or meaningfully about the LTBT and this is zero percent a claim about the LTBT, but as someone who has written a decent number of powerpoint decks that went to boards and used to be a management consultant and corporate strategy team member, I would generally be dissatisfied with the claim that a board’s most relevant metric is how many seats it currently has filled (so long as it has enough filled to meet quorum).

As just one example, it is genuinely way easier than you think for a board to have a giant binder full of “people we can emergency appoint to the board, if we really gotta” and be choosing not to exercise that binder because, conditional on no-emergency, they genuinely and correctly prefer waiting for someone being appointed to the board who has an annoying conflict that they’re in the process of resolving (e.g., selling off shares in a competitor or waiting out a post-government-employment “quiet period” or similar).

davekasten 2 Sep 2025 16:59 UTC
20 points
1
on: davekasten’s Shortform
I’m about to embark on the classic exercise of “think a bunch about AI policy.”

Does anyone actually have an up to date collection of “here are all the existing AI safety policy proposals out there”?

(Yes, I know, your existing proposal is already great and we should just implement it as-is. Think of the goal of this exercise being to convince someone else who needs to see a spreadsheet of “here are all the ideas, here is why idea number three is the best one”)

davekasten 11 Jul 2025 0:29 UTC
7 points
1
in reply to: JohnofCharleston’s comment on: Lessons from the Iraq War for AI policy
I think this is somewhat true, but also think in Washington it’s also about becoming known as “someone to go talk to about this” whether or not they’re your ally. Being helpful and genial and hosting good happy hours is surprisingly influential.

davekasten 11 Jul 2025 0:27 UTC
8 points
0
in reply to: JohnofCharleston’s comment on: Lessons from the Iraq War for AI policy
I agree with all of this—but also do think that there’s a real aspect here about some of the ideas lying around embedded existing policy constraints that were true both before and after the policy window changed. For example, Saudi Arabia was objectively a far better target for a 9/11-triggered casus belli than Iraq (15 of the 19 hijackers were Saudi citizens, as was bin Laden himself!), but no one had a proposal to invade Saudi Arabia on the shelf because in a pre-fracking United States, invading Saudi Arabia would essentially mean “shatter the US economy into a third Arab Oil Embargo.”

davekasten 11 Jul 2025 0:23 UTC
93 points
10
on: Lessons from the Iraq War for AI policy
I’m kind of confused by why these consequences didn’t hit home earlier.
I’m, I hate to say it, an old man among these parts in many senses; I voted in 2004, and a nontrivial percentage of the Lesswrong crowd wasn’t even alive then, and many more certainly not old enough to remember what it was like. The past is a different country, and 2004 especially so.
First: For whatever reason, it felt really really impossible for Democrats in 2004 to say that they were against the war, or that the administration had lied about WMDs. At the time, the standard reason why was that you’d get blamed for “not supporting the troops.” But with the light of hindsight, I think what was really going on was that we had gone collectively somewhat insane after 9/11 -- we saw mass civilian death on our TV screens happen in real time; the towers collapsing was just a gut punch. We thought for several hours on that day that several tens of thousands of people had died in the Twin Towers, before we learned just how many lives had been saved in the evacuation thanks to the sacrifice of so many emergency responders and ordinary people to get most people out.
And we wanted revenge. We just did. We lied to ourselves about WMDs and theories of regime change and democracy promotion, but the honest answer was that we’d missed getting bin Laden in Afghanistan (and the early days of that were actually looking quite good!), we already hated Saddam Hussein (who, to be clear, was a monstrous dictator), and we couldn’t invade the Saudis without collapsing our own economy. As Thomas Friedman put it, the message to the Arab world was “Suck on this.” And then we invaded Iraq, and collapsed their army so quickly and toppled their country in a month. And things didn’t start getting bad for months after, and things didn’t get truly awful until Bush’s second term. Heck, the Second Battle for Fallujah only started in November 2004.
And so, in late summer 2004, telling the American people that you didn’t support the people who were fighting the war we’d chosen to fight, the war that was supposed to get us vengeance and make us feel safe again—it was just not possible. You weren’t able to point to that much evidence that the war itself was a fundamentally bad idea, other than that some Europeans were mad at us, and we were fucking tired of listening to Europe. (Yes, I know this makes no sense, they were fighting and dying alongside us in Afghanistan. We were insane.)

Second: Kerry very nearly won—indeed, early on in election night 2004, it looked like he was going to! That’s part of why him losing was such a body blow to the Dems and, frankly, part of what opened up a lane for Obama in 2008. Perhaps part of why he ran it so close was that he avoided taking a stronger stance, honestly.

davekasten 28 Jun 2025 19:07 UTC
0 points
−5
in reply to: ryan_greenblatt’s comment on: Consider chilling out in 2028
I mean, two points:
1. We all work too many hours, working 70 hours a week persistently is definitely too many to maximize output. You get dumb fast after hour 40 and dive into negative productivity. There’s a robust organizational psych literature on this, I’m given to understand, that we all choose to ignore, because the first ~12 weeks or so, you can push beyond and get more done, but then it backfires.
2. You’re literally saying statements that I used to say before burning out, and that the average consultant or banker says as part of their path to burnout. And we cannot afford to lose either of you to burnout, especially not right now.
If you’re taking a full 4 weeks, great. 2 weeks a year is definitely not enough at a 70 hours a week pace, based on the observed long term health patterns of everyone I’ve known who works that pace for a long time. I’m willing to assert that you working 48/50ths of the hours a year you’d work otherwise is worth it, assuming fairly trivial speedups in productivity of literally just over 4% from being more refreshed, getting new perspectives from downing tools, etc.

davekasten 28 Jun 2025 19:01 UTC
23 points
7
on: Help the AI 2027 team make an online AGI wargame
I’d like to strongly assert that you’d want your design spec to be multiplayer from the start so that you can have virtually any arbitrary mix of LLMs and people. You’ll probably want this later and there are likely to be some design decisions that you’ll make wrong if you assume there’s never more than one person