TsviBT

Karma: 12,748

TsviBT 9 Jun 2026 8:22 UTC
9 points
8
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform

I think many people overrate how politically salient AI is.

I would say that much of my intuition that a pause (yes, driven by various national governments, and public desire) is plausibly doable comes from “the trajectory” of sentiments, rather than the total amount. I agree as a fraction of total political discourse it’s quite small.

Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep

This vaguely matches my impression for the most part, in the sense that if you look at discussion on AI in general, much of it may be negative but most of that isn’t people who deeply worry + care about x-risk. But to refine the picture, it’s mile wide, inch deep, but getting full of holes: it seems like there’s an ever growing number of people, including higher ups both in AI and also in government, who seem to take x-risk seriously, largely in words but also in more meaningful ways like drafting bills and stuff like that.

I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished.

Pardon my stupid question, but what goes wrong concretely? If you ban AGI, but let people keep running existing LLMs (say), does this really cause big and legible enough economic damage that voters would actually move? I mean, I’d think there’s lots of economically damaging things that don’t get credit-assigned into much actual vote shifts.

My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.

I may bow out, and feel free to take the maybe last, but I’ll just note that this still doesn’t make sense to me basically at all. Like, yes, there’s kinda-plausible scenarios where a global pause surprisingly ends up worse. But it would still be surprising, right? Like, there’s probably less human cloning right now, compared to the counterfactual where it wasn’t banned, right?? Someone could go underground with it, but that’s really hard and takes work!

I agree that slightly/somewhat unprecedentedly much access (official or espionage) might have to be somehow granted + enforced for treaty implementation. Maybe this is a strong defeater, I just don’t see it. Like, I agree it seems potentially kinda hard or quite hard in some scenarios. But this “global ban is actually worse than hugely resources companies going full tilt or even slightly less full tilt because of a “slowdown”″ seems galaxy-brained and false.

I’d also point out that a treaty isn’t a thing that happens, and then whatever’s written in the treaty determines how well the treaty helps the situation. A treaty is a step in a broader process that can continue to adapt and develop to avoid the dangerous stuff happening. Cf. https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad

TsviBT 8 Jun 2026 20:35 UTC
2 points
0
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform

people and governments taking AGI very seriously, something that is not the case right now and I believe is unlikely to become the case under a pause.

By a “pause”, the main thing I mean is “an international treaty to stop AGI globally”. I assume you’d think that’s unlikely because it’s unlikely that people + gvts would take it seriously enough. I don’t want to make a strong claim about it being likely feasible, and presumably even if it’s doable it would be a huge amount of hard work. But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?

we are very close to RSI/AGI/ASI in which case I think a global pause does not slow progress enough (due to enforcement issues)

I mean, I agree that it’s kinda hard, but if AGI is very close, it probably involves lots of compute, right? Big piles of compute seem plausibly regulable. That doesn’t seem like an infeasible enforcement issue.

we are not very close to RSI/AGI/ASI in which case we should hold off on a hard pause because the case for it will be stronger in the future. …. although I do worry that a stop would lower the salience of the issue.

Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?

TsviBT 8 Jun 2026 16:47 UTC
3 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform

Both of those posts have the form of “what would an agreement say” which I think is totally missing the hard part. So I think that points at the answer to your original question, and why others regard it as obvious and you do not.

You might be overinferring what I think these blog posts indicate? I’m just gesturing that I agree that the overall project of figuring out how the whole thing might be feasible is a worthy project.

The answer is “because there’s no political will”.

I know that this is a thing people say, and I agree there isn’t already automatically political will pre-gathered. But if the implication is that it would be an infeasible task to create and gather the political will for a global stop, that implication is one I strongly question! And so far I hear lots of signs pointing in the opposite direction, and grateful to the people working on that. I just wish that Anthropic would support those efforts.

WRT the Anthropic office visits: This has the general form of “it’s their fault not ours” which is suspicious.

Not blaming, describing. Can’t survive without describing.

(Anyway, just FYI, your time might be somewhat wasted if you want to get me on board with a particular approach / stance, because I’m much more commenting from the sidelines rather than an active participant; I’m focusing on other things, while others are actually working on communicating with the public and political leaders and so on.)

TsviBT 8 Jun 2026 4:44 UTC
4 points
4
in reply to: Seth Herd’s comment on: evhub’s Shortform

But I have heard people from the developer side of the fence say that they find LW a hostile environment and have trouble engaging here even though they feel they should. And the tone of discussions here certainly look like tribal dynamics and polarization are happening.

This makes sense. I will note however that when I (one time) asked an Anthropic employee about inviting someone over to their offices to explain / argue more in depth some crucial point (I forget; I think alignment difficulty), they said something like “last one or two times we tried that, the guest was dismissive”. So like, it looks a whole lot more like the crux is self-insulation, even if there is also undue hostility on LW. But, that is N=1. (I have other Ns that look like self-insulation, though of course that’s almost inherently ambiguous and my total N is small.)

(I do think in past I’ve at least watched from the sidelines, or even slightly participated in, arguably-undue dogpiley polite arguing, if not hostility.)

I won’t dive into that further right now, but I think it is a worthy collaborative project for LWers.

Definitely agree. (Cf. https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad and https://www.lesswrong.com/posts/X9Z9vdG7kEFTBkA6h/what-could-a-policy-banning-agi-look-like )

TsviBT 8 Jun 2026 4:32 UTC
2 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform
I agree re/ networks (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html). I also agree with polarization being bad. However:
- Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
- People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
I think this means something like, you strongly strongly criticize the behavior, but not demonize the person. I have for example said this: https://www.lesswrong.com/posts/CYTwRZtrhHuYf7QYu/a-case-for-courage-when-speaking-of-ai-danger?commentId=pLH6dxnTrTz56BQYj

I’m curious how else we-broadly can go about this better.

TsviBT 8 Jun 2026 4:11 UTC
6 points
0
in reply to: Seth Herd’s comment on: evhub’s Shortform

Framing it as you did without context looks somewhat combative to my eye,

I usually have a pretty bad intuitive reaction to people telling me that my writing is “too combative”, but I don’t know why fully, and in a small fraction of the cases I end up agreeing with them later, so I won’t respond further. I do want to say what I just said, though; not sure why, but maybe, for example, I feel that it’s somehow unfair, though since I can’t explain how this is a very unreliable sense.

and also seems like an isolated demand for rigor

I think it’s a really important and central supposition, and apparently has not been rigorously defended anywhere! This is 100% definitely not what the concept of “isolated demand for rigor” is for! What other such things am I allegedly not demanding rigor for, that I ought to be? Or do you disagree / am I confused somehow?

So I stand by my initial assessment of “concerning and probably more harm than good”.

Ok. (If you wanted to update me personally, you haven’t done that on this point.)

Showing proof of effort by laying out the reasons you’re asking make it clear that you’re not trolling.

Until this point, I have been genuinely unsure if there’s simply a report / blog post / something that someone might just link me to, explaining the case!! But yes, I think you’re right, I now agree it would be better for a comment like my first to come with a few sentences explaining that it seems like there isn’t such a case, that a global stop seems plausibly feasible, that Anthropic seems super far from appropriately supporting that, and that they should.

TsviBT 8 Jun 2026 4:00 UTC
17 points
20
in reply to: Seth Herd’s comment on: evhub’s Shortform
Ah ok here’s something that’s implicit, that I can now make explicit: IF you’re taking big actions (e.g. Anthropic doing frontier AGI research) AND your justifications for doing that rely on a quite unclear claim X, THEN you should have reasoned out X pretty well and defend it publicly OR ELSE you probably are deceiving yourself and/or others about why you’re doing what you’re doing. (There’s plenty of exceptions, but still.)

TsviBT 8 Jun 2026 3:31 UTC
15 points
2
in reply to: Seth Herd’s comment on: evhub’s Shortform
Sorry, I wasn’t fully explicit. When I said “Advocate for a global stop on AGI research”, I meant for us (broadly, including Evan, say) to advocate to world governments that they should institute a global stop, not that anyone’s supposed to persuade labs to stop. (I mean, separately I’d like to push on many points of intervention, including persuading AGI researchers to stop it.)

I still think your questions may include a degree of socratic trolling I think is ultimately unhelpful

I’m really not. Not sure why you think that. I can try to lay more things out, though I think I’ve indicated some / most of this? Anyway:
- I think it’s plausible that a global stop could be coordinated. I’m a layman, so I can’t, like, argue for this super well or coherently.
- I think that it’s bad for labs to be doing AGI research.
- Anthropic seems to justify pushing the frontier by saying a global pause is probably not doable.
- I have literally never heard a serious coherent case laid out that this is not doable. It’s just something that it’s embarrassing if you don’t understand. But that’s garbage reasoning.
- If I had to guess, I would guess that most of them (non-leadership Anthropic employees) broadly are just mindfucked somehow. However, I see plenty of bits of apparent good faith and good intentions, I know some of the people there and think they are smart and have or had good intentions, etc. Also, generally it seems rude to just assume there is no such case with checking. So, I have been trying to check by asking some Anthropic employees, and a bit of searching, and asking on LW. So far, no one has said “oh yeah here’s a link”, but people continue to argue or otherwise act as though it’s obvious. You tell me what I’m supposed to make of this! Maybe this is what comes off as trolling? Maybe we really should unban Said!

TsviBT 7 Jun 2026 20:08 UTC
14 points
0
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
1. You can do alignment research without doing current-flavored capabilities research. For example, no one has the concepts needed to understand values, which would probably be needed to do alignment; this can, and probably must, be investigated conceptually.
2. An intervention has many effects and differentially affects many processes, not just capabilities and alignment. Cf. https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#An_ontology_of_effects_of_interventions_on_world_processes For example, a stop gives everyone more time to understand the problem better and figure out ways to continue stopping. It also gives more time for human intelligence amplification, and other ways that humanity can make itself more able to survive the possibility of AGI.

TsviBT 7 Jun 2026 7:08 UTC
5 points
2
in reply to: JennaS’s comment on: evhub’s Shortform
I suspect this may be the case, but I’d like to know how people think about their beliefs about this question, but the way Anthropic employees (n=3ish) act when I try to ask about it is confusing. I wish they would just say “I’m relying on the judgement of so-and-so” or something like that, as it would be much clearer. (Cf. https://www.lesswrong.com/posts/zLG2DnJw6oEZqAgaE/tools-for-deferring-gracefully )

TsviBT 7 Jun 2026 1:45 UTC
LW: 3 AF: 1
0
AF
on: Views on when AGI comes and on strategy to reduce existential risk
I think this is a solid partial example of what I call confrontation-worthy empathy: https://www.youtube.com/watch?v=QYVOxn-Ndxw

TsviBT 7 Jun 2026 0:50 UTC
32 points
16
in reply to: Seth Herd’s comment on: evhub’s Shortform

I thought it was pretty clear why requirements to do more alignment work would be a lot easier and more attainable ask than a pause.

Well, Evan wrote:

I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable

It seems more attainable, sure, but also way way less useful, to the point where I don’t understand how it’s better than even something as sketchy as a napkin with “Advocate for a global stop on AGI research” written on it. Do you disagree?

We probably can’t get through a US government commitment to unilaterally pausing

I’m not especially advocating for that, though it might be good, IDK. I’m advocating for Anthropic to advocate for a global agreement.

I don’t know where anyone has tried to carefully make the case that a pause is impossible. It’s more that there’s an assumption that it’s quite hard, so the burden of proof falls more on those calling for one to propose a plan that could work. I don’t think a pause is impossible, although I think it’s unlikely to get one right now soon.

Do you agree that this assumption is a founding pillar of Anthropic’s strategy, at least as they present it? If this assumption is justifying doing frontier AGI research, should it be, like, argued somewhere? Don’t you think it’s weird that Anthropic as a whole big entity seems to be acting on this assumption, without a public argument for it? Do you think there’s no good argument for it, or do you think that it’s all in private somehow?

BTW, do you work at Anthropic?

TsviBT 6 Jun 2026 23:58 UTC
2 points
0
in reply to: Erich_Grunewald’s comment on: evhub’s Shortform
Feasible overall, so including political feasibility.

TsviBT 6 Jun 2026 20:22 UTC
44 points
35
in reply to: Seth Herd’s comment on: evhub’s Shortform

Surely you could give an elaborate answer yourself.

I really genuinely couldn’t!

I can list reasons that it’s difficult, and I can list reasons that it’s quite difficult in the long run, assuming compute isn’t a big bottleneck. But I don’t believe that the assumption of long-run, non-compute-bottlenecked research being the main driver of AGI progress is an assumption held by Evan or by Anthropic. I could be incorrect about this, happy to be corrected.

I don’t know it to be infeasible to globally prevent AGI research that involves significant resources. Do you know this? I’m genuinely asking. I’m not aware of any serious case laid out for this. That’s my ignorance, someone could just link such to me! (I did run an AI search for such, without relevant results.)

Like, in my ignorance, as far as I’ve personally seen / tracked, Anthropic seems be almost entirely, though not entirely, messaging on the presumption that a global stop is infeasible; and they use that presumption as justification for leading frontier AGI research including RSI. Is this presumption seriously defended anywhere?

TsviBT 6 Jun 2026 16:50 UTC
12 points
2
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
Is there a centralized explainer / synthesis / argument / whitepaper somewhere arguing a strong claim like “it’s quite likely infeasible to have a global strong slowdown / stop on AGI-ward research”?

TsviBT 6 Jun 2026 16:35 UTC
13 points
3
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform
Evan’s OP said:

I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible

I guess you’re taking this to apply specifically to the US, but not other places? Or is the idea something like:

Even with best efforts at global stop, there will be some places that continue. But they’ll go slower than the US. So we could slow down the US and other honorable regions, down to a rate that’s higher than the fastest dishonorable place but slower than currently.

Does this argument apply to nukes? Why / why not? How could espionage play a role in global enforcement?

I’m a bit skeptical (but quite unsure) that it’s very feasible to carry out significant AI research efforts without anyone knowing. How do you recruit scientists without anyone noticing, for example? E.g. if you approached a bunch of serious scientists to develop a serious bioweapon, I think / would hope (??) that you’d quickly get reported to something??

[Discourse note: I personally don’t have informed or strong opinions about this feasibility, short term or long term; what I’m confused about is the discourse that seems to assume a strong position that it’s not feasible, and I’m trying to check whether there’s a model behind that, or I’m missing something else, or what.]

TsviBT 6 Jun 2026 15:44 UTC
4 points
2
in reply to: Avi Brach-Neufeld’s comment on: evhub’s Shortform

Stopping all research would be much more difficulty politically in the U.S.

Why do you think this? (Where “all” is not 100% literal, but close to literal.) (What you say is plausible, but I don’t know it.)

As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.

Let’s assume it’s happening globally; or else, I wonder why that’s viewed as clearly infeasible.

TsviBT 6 Jun 2026 6:13 UTC
32 points
8
in reply to: evhub’s comment on: evhub’s Shortform
What’s wrong with just stopping research, period?

TsviBT 5 Jun 2026 3:01 UTC
2 points
0
on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition
Is the submission deadline set in stone, or will there be ongoing / rolling submissions? I was thinking of taking a minisabbatical to play with LLM stuff for things like this (in several different flavors). I also have a fair amount of experience doing factual lit research using LLMs as tools, but where getting accurate answers requires nontrivial workflows. But July 19 is too soon for me. Just a datapoint.

TsviBT 4 Jun 2026 3:01 UTC
6 points
0
in reply to: Zack_M_Davis’s comment on: Comment on “Banning Said Achmiz”
I think your “go down this rabbit hole where you’re not actually trying to talk about” story could potentially fit the improv game framework, actually?

Absolutely not, no. I hope the following can help explain.

you should have more clearly flagged that you were commenting in an “adversarial” role rather than a “cooperative” one with respect to my research program

Yes, IMO this is a much more prosocial pattern and tends to go better for everyone involved. You’re saying this is “controlling”, but again, there’s a simple matter of matching intents. Like a traffic light.

Suppose this happens:

I write a big post that assumes X and goes on to discuss Y. Then you come in and say “It seems like this post assumes X; Is that right? Maybe this is a bit off topic / off of your interests, but I think that X is false, and I think this affects/invalidates your conclusions about Y in reality.”.

Now, I could be annoyed or something that I’m not getting the engagement I want, or annoyed that you’re getting more attention, or something. But I think in this case that annoyance is probably largely my problem, or possibly a problem with the local discourse [i.e. collective user behavior] (e.g. because it’s not able to allocate its cognitive resources properly). It’s a perfectly valid way for you to engage in discourse in a public forum; it’s interesting + relevant to discuss whether X is true, whether it affects my conclusions about Y, and whether my post does assume X.

However, suppose you come in and make an ambiguous comment, e.g. “just asking questions”, while in fact you mostly already have an opinion which you are for some reason not stating. Specifically, highly ambiguous about what you care about / what you’re trying to talk about / what topic you’re trying to get at. (And perhaps, further, the comment is elliptical, or seems to presume something about what I said incorrectly, or seems to signal that I “ought” to engage or have a response or something.) Well, now I might want to engage on the assumption that you’re talking about Y. There might also be attentional presumptions that I ought to engage. For example, it could be that most readers / the community broadly agrees that X is true, and doesn’t view it as that worthwhile to debate whether X is true unless there’s important new evidence. But if your comment is ambiguous, then I as the author and especially the audience would not have common knowledge that you’re not arguing against my claims about Y (presuming X).

If you signpost what you’re doing, then IMO what’s supposed to happen is this:
- If your arguments are bad, you get ignored.
- If your arguments are kind good / interesting, you get upvoted and discussed, but I may ignore because I believe X.
- If your arguments are really good, then I’m eventually “forced” (within some ideal discourse community) to grapple with what you’re saying somehow.
This seems to be like both feasible and a much better sort of attentional traffic-light coordination. (Though plausibly I’m confused / missing important stuff, and I’m confident this is far from comprehensively addressing the overall conflict at hand.)

As a random illustration (forgive my poor memory), for something like a year or two (? not sure), I had various “debates / debate skirmishes” with various other people doing agent foundations research at MIRI (circa 2019-2021 I guess, though I’m quite fuzzy on that). This would generally take the form of me saying stuff like “this whole research program is dooomed because it is too slow / it looks at the wrong data / it’s asking the wrong questions / it’s presuming away too much of the important stuff / it’s not sufficiently grappling with our lack of the appropriate concepts / etc.” or similar, and then trying to communicate about that, mostly by waving my arms around. IDK how much of an impression it made, but anyway, eventually I realized that I and the other researchers just had “fundamentally” different background presumptions about how to approach the whole problem of alignment / research taste / how to think / what questions to ask / how important methodology is / etc.; and therefore they weren’t trying to orient to the same things I was trying to orient towards; and therefore it was somewhat of a waste of effort to invest too much (beyond more initial fun versions) in these debates, and generally stop trying to attack the core parts of the problem together as a team with them. If I had been able to more quickly figure out, and state explicitly, these fundamental positions / orientations, then I could have avoided derailing other people’s research conversations as much. (I don’t think this is a direct analogy in most respects to the topic; my intent is to exemplify the structure of “unexpressed basic difference of orientation --> wasted effort; expressed basic difference --> more efficiently parallel / independent processing” or something like that. My guess is that in the cases at hand, it would be much easier to just state the disagreement directly, though maybe that’s incorrect?)