TsviBT

Karma: 12,668

TsviBT 8 Jun 2026 4:44 UTC
2 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform

But I have heard people from the developer side of the fence say that they find LW a hostile environment and have trouble engaging here even though they feel they should. And the tone of discussions here certainly look like tribal dynamics and polarization are happening.

This makes sense. I will note however that when I (one time) asked an Anthropic employee about inviting someone over to their offices to explain / argue more in depth some crucial point (I forget; I think alignment difficulty), they said something like “last one or two times we tried that, the guest was dismissive”. So like, it looks a whole lot more like the crux is self-insulation, even if there is also undue hostility on LW. But, that is N=1. (I have other Ns that look like self-insulation, though of course that’s almost inherently ambiguous and my total N is small.)

(I do think in past I’ve at least watched from the sidelines, or even slightly participated in, arguably-undue dogpiley polite arguing, if not hostility.)

I won’t dive into that further right now, but I think it is a worthy collaborative project for LWers.

Definitely agree. (Cf. https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad and https://www.lesswrong.com/posts/X9Z9vdG7kEFTBkA6h/what-could-a-policy-banning-agi-look-like )

TsviBT 8 Jun 2026 4:32 UTC
2 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform
I agree re/ networks (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html). I also agree with polarization being bad. However:
- Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
- People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
I think this means something like, you strongly strongly criticize the behavior, but not demonize the person. I have for example said this: https://www.lesswrong.com/posts/CYTwRZtrhHuYf7QYu/a-case-for-courage-when-speaking-of-ai-danger?commentId=pLH6dxnTrTz56BQYj

I’m curious how else we-broadly can go about this better.

TsviBT 8 Jun 2026 4:11 UTC
4 points
2
in reply to: Seth Herd’s comment on: evhub’s Shortform

Framing it as you did without context looks somewhat combative to my eye,

I usually have a pretty bad intuitive reaction to people telling me that my writing is “too combative”, but I don’t know why fully, and in a small fraction of the cases I end up agreeing with them later, so I won’t respond further. I do want to say what I just said, though; not sure why, but maybe, for example, I feel that it’s somehow unfair, though since I can’t explain how this is a very unreliable sense.

and also seems like an isolated demand for rigor

I think it’s a really important and central supposition, and apparently has not been rigorously defended anywhere! This is 100% definitely not what the concept of “isolated demand for rigor” is for! What other such things am I allegedly not demanding rigor for, that I ought to be? Or do you disagree / am I confused somehow?

So I stand by my initial assessment of “concerning and probably more harm than good”.

Ok. (If you wanted to update me personally, you haven’t done that on this point.)

Showing proof of effort by laying out the reasons you’re asking make it clear that you’re not trolling.

Until this point, I have been genuinely unsure if there’s simply a report / blog post / something that someone might just link me to, explaining the case!! But yes, I think you’re right, I now agree it would be better for a comment like my first to come with a few sentences explaining that it seems like there isn’t such a case, that a global stop seems plausibly feasible, that Anthropic seems super far from appropriately supporting that, and that they should.

TsviBT 8 Jun 2026 4:00 UTC
4 points
2
in reply to: Seth Herd’s comment on: evhub’s Shortform
Ah ok here’s something that’s implicit, that I can now make explicit: IF you’re taking big actions (e.g. Anthropic doing frontier AGI research) AND your justifications for doing that rely on a quite unclear claim X, THEN you should have reasoned out X pretty well and defend it publicly OR ELSE you probably are deceiving yourself and/or others about why you’re doing what you’re doing. (There’s plenty of exceptions, but still.)

TsviBT 8 Jun 2026 3:31 UTC
4 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform
Sorry, I wasn’t fully explicit. When I said “Advocate for a global stop on AGI research”, I meant for us (broadly, including Evan, say) to advocate to world governments that they should institute a global stop, not that anyone’s supposed to persuade labs to stop. (I mean, separately I’d like to push on many points of intervention, including persuading AGI researchers to stop it.)

I still think your questions may include a degree of socratic trolling I think is ultimately unhelpful

I’m really not. Not sure why you think that. I can try to lay more things out, though I think I’ve indicated some / most of this? Anyway:
- I think it’s plausible that a global stop could be coordinated. I’m a layman, so I can’t, like, argue for this super well or coherently.
- I think that it’s bad for labs to be doing AGI research.
- Anthropic seems to justify pushing the frontier by saying a global pause is probably not doable.
- I have literally never heard a serious coherent case laid out that this is not doable. It’s just something that it’s embarrassing if you don’t understand. But that’s garbage reasoning.
- If I had to guess, I would guess that most of them (non-leadership Anthropic employees) broadly are just mindfucked somehow. However, I see plenty of bits of apparent good faith and good intentions, I know some of the people there and think they are smart and have or had good intentions, etc. Also, generally it seems rude to just assume there is no such case with checking. So, I have been trying to check by asking some Anthropic employees, and a bit of searching, and asking on LW. So far, no one has said “oh yeah here’s a link”, but people continue to argue or otherwise act as though it’s obvious. You tell me what I’m supposed to make of this! Maybe this is what comes off as trolling? Maybe we really should unban Said!

TsviBT 7 Jun 2026 20:08 UTC
14 points
0
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
1. You can do alignment research without doing current-flavored capabilities research. For example, no one has the concepts needed to understand values, which would probably be needed to do alignment; this can, and probably must, be investigated conceptually.
2. An intervention has many effects and differentially affects many processes, not just capabilities and alignment. Cf. https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#An_ontology_of_effects_of_interventions_on_world_processes For example, a stop gives everyone more time to understand the problem better and figure out ways to continue stopping. It also gives more time for human intelligence amplification, and other ways that humanity can make itself more able to survive the possibility of AGI.

TsviBT 7 Jun 2026 7:08 UTC
5 points
2
in reply to: JennaS’s comment on: evhub’s Shortform
I suspect this may be the case, but I’d like to know how people think about their beliefs about this question, but the way Anthropic employees (n=3ish) act when I try to ask about it is confusing. I wish they would just say “I’m relying on the judgement of so-and-so” or something like that, as it would be much clearer. (Cf. https://www.lesswrong.com/posts/zLG2DnJw6oEZqAgaE/tools-for-deferring-gracefully )

TsviBT 7 Jun 2026 1:45 UTC
LW: 2 AF: 1
0
AF
on: Views on when AGI comes and on strategy to reduce existential risk
I think this is a solid partial example of what I call confrontation-worthy empathy: https://www.youtube.com/watch?v=QYVOxn-Ndxw

TsviBT 7 Jun 2026 0:50 UTC
32 points
16
in reply to: Seth Herd’s comment on: evhub’s Shortform

I thought it was pretty clear why requirements to do more alignment work would be a lot easier and more attainable ask than a pause.

Well, Evan wrote:

I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable

It seems more attainable, sure, but also way way less useful, to the point where I don’t understand how it’s better than even something as sketchy as a napkin with “Advocate for a global stop on AGI research” written on it. Do you disagree?

We probably can’t get through a US government commitment to unilaterally pausing

I’m not especially advocating for that, though it might be good, IDK. I’m advocating for Anthropic to advocate for a global agreement.

I don’t know where anyone has tried to carefully make the case that a pause is impossible. It’s more that there’s an assumption that it’s quite hard, so the burden of proof falls more on those calling for one to propose a plan that could work. I don’t think a pause is impossible, although I think it’s unlikely to get one right now soon.

Do you agree that this assumption is a founding pillar of Anthropic’s strategy, at least as they present it? If this assumption is justifying doing frontier AGI research, should it be, like, argued somewhere? Don’t you think it’s weird that Anthropic as a whole big entity seems to be acting on this assumption, without a public argument for it? Do you think there’s no good argument for it, or do you think that it’s all in private somehow?

BTW, do you work at Anthropic?

TsviBT 6 Jun 2026 23:58 UTC
2 points
0
in reply to: Erich_Grunewald’s comment on: evhub’s Shortform
Feasible overall, so including political feasibility.

TsviBT 6 Jun 2026 20:22 UTC
42 points
33
in reply to: Seth Herd’s comment on: evhub’s Shortform

Surely you could give an elaborate answer yourself.

I really genuinely couldn’t!

I can list reasons that it’s difficult, and I can list reasons that it’s quite difficult in the long run, assuming compute isn’t a big bottleneck. But I don’t believe that the assumption of long-run, non-compute-bottlenecked research being the main driver of AGI progress is an assumption held by Evan or by Anthropic. I could be incorrect about this, happy to be corrected.

I don’t know it to be infeasible to globally prevent AGI research that involves significant resources. Do you know this? I’m genuinely asking. I’m not aware of any serious case laid out for this. That’s my ignorance, someone could just link such to me! (I did run an AI search for such, without relevant results.)

Like, in my ignorance, as far as I’ve personally seen / tracked, Anthropic seems be almost entirely, though not entirely, messaging on the presumption that a global stop is infeasible; and they use that presumption as justification for leading frontier AGI research including RSI. Is this presumption seriously defended anywhere?

TsviBT 6 Jun 2026 16:50 UTC
10 points
0
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
Is there a centralized explainer / synthesis / argument / whitepaper somewhere arguing a strong claim like “it’s quite likely infeasible to have a global strong slowdown / stop on AGI-ward research”?

TsviBT 6 Jun 2026 16:35 UTC
13 points
3
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
Evan’s OP said:

I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible

I guess you’re taking this to apply specifically to the US, but not other places? Or is the idea something like:

Even with best efforts at global stop, there will be some places that continue. But they’ll go slower than the US. So we could slow down the US and other honorable regions, down to a rate that’s higher than the fastest dishonorable place but slower than currently.

Does this argument apply to nukes? Why / why not? How could espionage play a role in global enforcement?

I’m a bit skeptical (but quite unsure) that it’s very feasible to carry out significant AI research efforts without anyone knowing. How do you recruit scientists without anyone noticing, for example? E.g. if you approached a bunch of serious scientists to develop a serious bioweapon, I think / would hope (??) that you’d quickly get reported to something??

[Discourse note: I personally don’t have informed or strong opinions about this feasibility, short term or long term; what I’m confused about is the discourse that seems to assume a strong position that it’s not feasible, and I’m trying to check whether there’s a model behind that, or I’m missing something else, or what.]

TsviBT 6 Jun 2026 15:44 UTC
4 points
2
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform

Stopping all research would be much more difficulty politically in the U.S.

Why do you think this? (Where “all” is not 100% literal, but close to literal.) (What you say is plausible, but I don’t know it.)

As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.

Let’s assume it’s happening globally; or else, I wonder why that’s viewed as clearly infeasible.

TsviBT 6 Jun 2026 6:13 UTC
32 points
9
in reply to: evhub’s comment on: evhub’s Shortform
What’s wrong with just stopping research, period?

TsviBT 5 Jun 2026 3:01 UTC
2 points
0
on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition
Is the submission deadline set in stone, or will there be ongoing / rolling submissions? I was thinking of taking a minisabbatical to play with LLM stuff for things like this (in several different flavors). I also have a fair amount of experience doing factual lit research using LLMs as tools, but where getting accurate answers requires nontrivial workflows. But July 19 is too soon for me. Just a datapoint.

TsviBT 4 Jun 2026 3:01 UTC
6 points
0
in reply to: Zack_M_Davis’s comment on: Comment on “Banning Said Achmiz”
I think your “go down this rabbit hole where you’re not actually trying to talk about” story could potentially fit the improv game framework, actually?

Absolutely not, no. I hope the following can help explain.

you should have more clearly flagged that you were commenting in an “adversarial” role rather than a “cooperative” one with respect to my research program

Yes, IMO this is a much more prosocial pattern and tends to go better for everyone involved. You’re saying this is “controlling”, but again, there’s a simple matter of matching intents. Like a traffic light.

Suppose this happens:

I write a big post that assumes X and goes on to discuss Y. Then you come in and say “It seems like this post assumes X; Is that right? Maybe this is a bit off topic / off of your interests, but I think that X is false, and I think this affects/invalidates your conclusions about Y in reality.”.

Now, I could be annoyed or something that I’m not getting the engagement I want, or annoyed that you’re getting more attention, or something. But I think in this case that annoyance is probably largely my problem, or possibly a problem with the local discourse [i.e. collective user behavior] (e.g. because it’s not able to allocate its cognitive resources properly). It’s a perfectly valid way for you to engage in discourse in a public forum; it’s interesting + relevant to discuss whether X is true, whether it affects my conclusions about Y, and whether my post does assume X.

However, suppose you come in and make an ambiguous comment, e.g. “just asking questions”, while in fact you mostly already have an opinion which you are for some reason not stating. Specifically, highly ambiguous about what you care about / what you’re trying to talk about / what topic you’re trying to get at. (And perhaps, further, the comment is elliptical, or seems to presume something about what I said incorrectly, or seems to signal that I “ought” to engage or have a response or something.) Well, now I might want to engage on the assumption that you’re talking about Y. There might also be attentional presumptions that I ought to engage. For example, it could be that most readers / the community broadly agrees that X is true, and doesn’t view it as that worthwhile to debate whether X is true unless there’s important new evidence. But if your comment is ambiguous, then I as the author and especially the audience would not have common knowledge that you’re not arguing against my claims about Y (presuming X).

If you signpost what you’re doing, then IMO what’s supposed to happen is this:
- If your arguments are bad, you get ignored.
- If your arguments are kind good / interesting, you get upvoted and discussed, but I may ignore because I believe X.
- If your arguments are really good, then I’m eventually “forced” (within some ideal discourse community) to grapple with what you’re saying somehow.
This seems to be like both feasible and a much better sort of attentional traffic-light coordination. (Though plausibly I’m confused / missing important stuff, and I’m confident this is far from comprehensively addressing the overall conflict at hand.)

As a random illustration (forgive my poor memory), for something like a year or two (? not sure), I had various “debates / debate skirmishes” with various other people doing agent foundations research at MIRI (circa 2019-2021 I guess, though I’m quite fuzzy on that). This would generally take the form of me saying stuff like “this whole research program is dooomed because it is too slow / it looks at the wrong data / it’s asking the wrong questions / it’s presuming away too much of the important stuff / it’s not sufficiently grappling with our lack of the appropriate concepts / etc.” or similar, and then trying to communicate about that, mostly by waving my arms around. IDK how much of an impression it made, but anyway, eventually I realized that I and the other researchers just had “fundamentally” different background presumptions about how to approach the whole problem of alignment / research taste / how to think / what questions to ask / how important methodology is / etc.; and therefore they weren’t trying to orient to the same things I was trying to orient towards; and therefore it was somewhat of a waste of effort to invest too much (beyond more initial fun versions) in these debates, and generally stop trying to attack the core parts of the problem together as a team with them. If I had been able to more quickly figure out, and state explicitly, these fundamental positions / orientations, then I could have avoided derailing other people’s research conversations as much. (I don’t think this is a direct analogy in most respects to the topic; my intent is to exemplify the structure of “unexpressed basic difference of orientation --> wasted effort; expressed basic difference --> more efficiently parallel / independent processing” or something like that. My guess is that in the cases at hand, it would be much easier to just state the disagreement directly, though maybe that’s incorrect?)

TsviBT 4 Jun 2026 1:24 UTC
6 points
4
in reply to: Zack_M_Davis’s comment on: Comment on “Banning Said Achmiz”
[NB: I disagree voted but not karma voted.]

I think it would be gullible to uncritically assume all self reports of hurt feelings are not covering up some sort of bad behavior. I don’t think that’s what you’re saying, but I’m not actually sure what you’re actually saying. My best guess literal reading of what you wrote is that you’re saying that all these self-reports are more or less always (or usually? or what?) mainly motivated by a kind of coverup for poor reasoning. If you’re saying this, then I probably pretty strongly disagree. It could depend on which people we’re talking about, but I think there are other frequent main reasons for someone to end up with that report, such as the one I sketched in my example.

TsviBT 4 Jun 2026 0:12 UTC
18 points
13
in reply to: Eli Tyre’s comment on: Comment on “Banning Said Achmiz”
(Not that that’s a bad idea, but regarding both this idea and my original comment:)

My guess is that there’s core aspects of the Said dispute (as far as I understand it, not really being read in) that Crocker’s rules and butterfly ideas doesn’t address—issues around coordination and cognitive labor.

As an example to illustrate the type of thing I’m gesturing at:

Suppose I start talking about X, and you are interested in Y. Then you ask me elliptical questions that could ambiguously be “about X” (in the sense of having the Gricean mental context of investigating X together), causing me to go down this rabbit hole where you’re not actually trying to talk about X.

This can be construed as a kind of defection against the conversational project of even talking about the same thing.

This can be largely orthogonal to butterfly/Crocker (if those are properly opposed). E.g. you could be pretty polite while derailing. And conversely, in some contexts you can be “Crockerish” productively and normatively even with butterfly ideas. For example, quickly coming up with counterexamples to a conjectured generalization, disregarding hopes about the hypothesis being true, can still be a cooperative butterfly sanctuary—because, for example, you are not being dismissive in tone, and are devoting attention to the actual butterfly idea, and are rolling with frequent revisions of the conjecture without getting snippy.

(A further random guess I have, again not being read in including not having fully read the OP, would be that talk of “hurt feelings” in this whole context is confused about some of the cases, because the feelings aren’t “you said mean things about me” but more like “I turned out to be frustrated with this thread, and even if I can’t explain exactly what’s happening I still think it was kinda adversarial and a regretted use of my time, which is an annoying situation and I just want to avoid this.”, or something like that. In other words, the thing that’s going wrong in some of these cases might be NOT something that Crocker’s rules are even supposed to be opting you into.)

TsviBT 31 May 2026 0:07 UTC
16 points
6
in reply to: Steven Byrnes’s comment on: Comment on “Banning Said Achmiz”
A further wrinkle is that one man’s Gricean failure is another man’s crusade to uphold norms. Elliptical comments leave a whole lot of ambiguity (e.g. are you curious, or critiquing; do you think you already know how to better do what I was trying and allegedly failing to do), and principles of charity fall prey to ambiguity by regrettedly overinvesting effort.