Thanks for the feedback, good to know.
Davidmanheim
Karma: 6,477
The problem with Federalism isn’t that it doesn’t capture the idea, it’s that it also requires stable delegation and authority, which this post really isn’t about. It’d partly subsidiarity, or autonomy, but also state capacity mismatch, and Fukuyama’s discussions of decay, and institutional drift, or maybe closer to fragility of complex systems, and the way they fail? (Charles Perrow’s work, specifically.)
But I think you’re talking about it differently than any of those alone, and none have a simple term for this.
That’s a really good point.
I think it’s selection bias on the conspiracies finding those that are the most plausible or most successful, and accurate perception of how poorly the examples you’ve seen personally work out.
I don’t think the terminology was clear. (I finished 100% of the essay and got to this comment before I understood why you picked the word.)
In different words, if the garden is too large to defend...
Defense of territory is linear with border area, defense against internal threats is linear in area, defense against org scaling failures seems at least linear in the (already exponentially growing) size. So yes, EA has a huge problem with scaling more, and more, and more. This is mostly because scaling is very, very hard.
I think I lost your point now, this got very meta.
The thing that I find confusing is I feel like I get the vibe that is meant, but I don’t understand what the brightline or criteria are for when something is acceptable vs violates some norm.
I don’t think there is a bright line, there’s just a point being made about a gradient where discourse chess is on one side, and talking about object level facts in on the other. And I pointed out that on the chess side, people suck at getting what they want.
Similarly for the other points, I don’t really care about exact lines, I’m not being prescriptive.
I’m not really sure how to take your statement as not being an accusation of EAs doing the thing you are criticizing.
You should read the last paragraph of my response. I don’t need to ask what prominent EAs would think, since I know what they did, in fact, say. And you may not have been around, but if you look at older posts on the EA forum, this wasn’t exactly a secret, it was a declared intention. And looking back, Rob was complaining about this 5 years ago. And here’s complaining about related issues 7 years ago. So I agree that imputing this type of behavior, as an accusation, is worrying, but that’s different than pointing out when the behavior was in fact intentional.
I am stating, not implying, that the galaxy brain plan for stretching the Overton window backfired on them. But I’m not criticizing the position, I’m criticizing the efficacy of the strategy. Whether it was a luxury belief seems like a mostly separate point, though correlated in that such clever strategies do seem to be more likely to be pursued by the educated and wealthy.
I’m somewhat confused by what people mean by “strategic” in these discussions.
Apologies, I tried to make this clear: I am referring to “high-dimensional discourse chess” that requires asserting or assuming “we can model how public acceptability shifts and cleverly intervene to steer those shifts.” That’s not about communicating an idea, it’s about the goal of convincing people of something in order to have them react in order to change the pubic acceptability of another thing.
Let’s take the “DEFUND THE POLICE” example from the post.
Of course, not everyone who said “Defund the police” was playing games—some were true extremists, including leftist anarchists. But far more were being duped by those trying to play those games, advocating for what they thought was good based on social proof, almost always without a coherent replacement in their mind. That is, they generally weren’t advocating anarchy, they were advocating replacing police with something else undefined. It seems you are saying something similar; you dislike the current system, and say it should be replaced instead of reformed, but don’t have a clear argument for the details.
I think the best approach is to… leave aiside the dicussions about whether someone is being stategic, how truth-seeking they are, if they are “gaslighting” etc.
I agree—and if anything, think it’s exactly in line with the post’s conclusion? Specifically, in the post, I don’t argue that readers should dismiss anyone else’s views for playing discourse games, I argue that they should not attempt them.
The only reason I feel comfortable using the Defund the Police example is because the leaders of the movement were explicit in their intent to widen the window. For example, “Progressive Congresswoman Alexandria Ocasio-Cortez called the use of language like “defund” an “excellent choice” for those who have been trying for years to “prompt a national conversation” about police. “‘Refund’ or ‘reallocate’ didn’t do that,” she tweeted. The shocking language of the slogans can help shift the Overton Window, making what might otherwise be politically controversial interventions more palatable. ”
And I wasn’t guessing about EA either; I have been in the room, repeatedly, when senior people in EA talked about shifting the Overton window on AI risk. So yes, don’t accuse others of doing this, but that doesn’t mean you can’t call them out when they say they are doing it!
Many of those groups would have been left to starve to death a century ago. The idea that a rising tide doesn’t lift all boats is historically illiterate, and so you would need a more specific argument for it to make sense here in the case of (non-omnicidal) ASI.
Regardless of the question of AI impacts, and meaning, it’s simply untrue that greater wealth hasn’t changed the picture tremendously. Life expectancy has shot up. The idea that being richer didn’t matter is bullshit. Yes, it costs $5,000 to save a life today, and we haven’t saturated those opportunities. But in 1970, 55 years ago, we were in the middle of smallpox eradication, which like had a cost per life saved likely under $1,000, in current dollars. Oral rehydration therapy was becoming available, and was not yet widespread; it also likely had a cost in a similar range per life saved. And going back further, in the 1920s, Russia was having a massive famine. In 1921–23 there was a relief effort; the American Relief Administration probably saved several million lives as a cost of around $300 per life saved in current dollars.
I don’t think that advice is obviously wrong—the costs of being outside are real, and so there are definitely times when staying inside the Overton window is important. Not alienating policymakers seems like a good place to simply not talk about certain things, at least sometimes. (c.f. Eliezer’s Time article being laughed at during the White House press briefing. That worked out fine, Eliezer wasn’t trying to stay in the good graces of policymakers. Someone from CSET writing the same would have been a mistake.)
So it’s further evidence that EAs are sensitive to the questions, but not an example of what I think Rob was criticizing, and as noted, not something I think was the wrong call.
You Aren’t in Charge of the Overton Window; Politics Is Not Interior Design
Treaties, Regulations, and Research can be Complements
Fantastic work, very happy to see concrete data collection like this!
Minor methodology note:
the US as a whole is way less dense than NYC so going to less dense neighborhoods is likely more representative
That doesn’t work; less dense in NYC generally means significantly wealthier, and NYC already skews much wealthier than average.
It did not take long at all for experts to catch up to the prediction: https://sockpuppet.org/blog/2026/03/30/vulnerability-research-is-cooked/
Within the next few months, coding agents will drastically alter both the practice and the economics of exploit development. Frontier model improvement won’t be a slow burn, but rather a step function. Substantial amounts of high-impact vulnerability research (maybe even most of it) will happen simply by pointing an agent at a source tree and typing “find me zero days”.
You definitely get epistemic points for attempting alternative theories, but no additional points, I think, for providing ones that seem plausibly as explanatory as the original ones provided.
There are two different arguments that it’s important not to conflate, and which I did not do a good enough job of clarifying. The first, which you are talking about in #1, is that the incentives are such that countries might not sign a given treaty. I think this is true in the real world primarily because people are failing to understand the risk, and secondarily because the dynamics if ASI isn’t a threat are such that a race would make sense. The second, which you kind-of are talking about in #2, is whether the treaty will work to stop the creation of misaligned ASI. That goal was reasonably clear in the post, I think, and yes, I do think it is the case that rational actors would agree to mutually beneficial not-everyone-dying treaties, and think that it’s obvious that such treaties are possible to specify. To the extent that countries fall short of being rational actors, this goes back to question #1, which, again, I didn’t clearly distinguish but wasn’t trying to address—but IABIED was.
If you’re entirely uncertainty insensitive and you’re looking at a distribution over final outcomes of the world rather than a specific compound variable, sure. But that’s almost never the case. I’m not saying that you should be naively taking the expectation of poorly thought through distributions, but it’s insane to oppose sensitivity analysis.
(Also, unlike in your example where you impose uniform uncertainties on the powers of the multiplied variables instead of on the variables themselves, it’s actually impossible to have variables that themselves have reasonably quantified distributions with means that match the expectation and then get a different expectation because you multiply them together. But I agree that what people often actually do is make stupid mistakes that break this.)
...but then you don’t extrapolate to tat trend going even further with more wealth?
Yes, AGI will be default be a disaster, but in the cases where it is not, it is hard for me to understand the argument here that everyone would be better off by default, that we couldn’t end up benefitting greatly. I don’t disagree with most of the piece, but this part seemed hard to beleive.