Here’s a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):
Enforce that labs must spend at least X% of their compute on:
Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won’t be as relevant to the labs’ specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research the lab tries to do via this mechanism much less likely to advantage them differentially (though of course they could still just end up doing bad safety research if they pick bad external safety orgs).
Internal safety research that the lab makes fully public. Again I think this substantially disincentivizes capabilities research, since any capabilities research that is fully public will be much less able to differentially advantage that lab specifically, as others can just copy it (though again of course the labs could still just do bad public safety research).
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible—and it also differentially advantages alignment research by choosing conditions (external or fully public research) that make it much harder to do capabilities under those conditions but still pretty easy to do alignment. If you just say that compute has to go to some sort of nebulous “safety” purpose, that could easily be gamed, but I think the conditions above do a pretty good job of operationalizing criteria that are fairly hard to game. And even if you do get some gaming, on the margin you are still making capabilities research more costly, and thus buying yourself some additional slowdown, as well as making alignment research less costly, and thus buying yourself some additional alignment research.
(A possible second-best proposal if this is unworkable for some reason is just to enforce X% of compute go to external inference of some variety, which is more concrete and still gets you lots of slowdown, but less differentially advantages alignment research.)
If you think that alignment is somewhat tractable and that any sort of pause won’t be able to constrain 100% of labs then you are just buying yourself a finite amount of time. You then have to actually do safety research during that time.
Stopping all research would be much more difficulty politically in the U.S.
As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.
If you are very confident that alignment is very very hard and think that any warning signs found during the additional safety research will definitely be ignored, I agree this doesn’t do much.
I think nuclear non-proliferation is a good benchmark here. The largest powers really cared about non-proliferation and did a good, but not perfect job (we’ve had four nations become nuclear since the NPT). I think many people overrate how difficult it would be to monitor/constrain AI research because of how specialized the GPU supply chain is, but I still think it’s harder to monitor than nuclear weapon development (Probably, I haven’t read up on this a lot).
Non-proliferation is not a perfect analogy for different reasons, but I think those reasons push in different directions and would need a strong argument on why we would do better with AI.
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible
I guess you’re taking this to apply specifically to the US, but not other places? Or is the idea something like:
Even with best efforts at global stop, there will be some places that continue. But they’ll go slower than the US. So we could slow down the US and other honorable regions, down to a rate that’s higher than the fastest dishonorable place but slower than currently.
Does this argument apply to nukes? Why / why not? How could espionage play a role in global enforcement?
I’m a bit skeptical (but quite unsure) that it’s very feasible to carry out significant AI research efforts without anyone knowing. How do you recruit scientists without anyone noticing, for example? E.g. if you approached a bunch of serious scientists to develop a serious bioweapon, I think / would hope (??) that you’d quickly get reported to something??
[Discourse note: I personally don’t have informed or strong opinions about this feasibility, short term or long term; what I’m confused about is the discourse that seems to assume a strong position that it’s not feasible, and I’m trying to check whether there’s a model behind that, or I’m missing something else, or what.]
First a clarification: In your initial post I took “stopping research” to mean stopping all AI research without a safety carve out. If that’s not the case please correct me.
Let’s say any policy intervention has a capabilities progress multiplier (c). So if at our current rate we are x years from developing powerful AI (defined as having a better than 50% chance of being able to take over the world for it’s creator if aligned or for itself if not aligned) then it would take x/c years under that policy. Let’s also think of an safety progress multiplier (s) so that in y/s years we will be confident we can safely align powerful AI where y is how far we are at the current rate.
We want a policy that either makes x/c so large we don’t care about it or makes x/c>y/s. I need to do more research, but my initial though is that making y/c so large we don’t care about it is very hard. If making y/c so large we don’t care about it seems plausible to you, we can get into the details more. I have some intuitions here, but need to do more research.
Assuming we are not on track to solve alignment (which I think we would both agree is likely true) then we know x<y. Thus any policy intervention needs to have s>c. I think that is true of Evan’s policy, but not true of a global moratorium (because defectors are likely to differentially prioritize capabilities over research).
You can do alignment research without doing current-flavored capabilities research. For example, no one has the concepts needed to understand values, which would probably be needed to do alignment; this can, and probably must, be investigated conceptually.
Before we get to these points I want to rehilight and briefly expand on the other two points in my original comments.
Stopping all research would be much more difficulty politically in the U.S.
This would erase trillions of dollars of value, potentially 10s of trillions, something the U.S. government has a lot of incentives not to do.
As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.
To get anything close to a real global pause the U.S. or others would have to commit (and probably take) military action to enforce it, again something the U.S. government has a lot of incentives not to do.
To address your points. I agree with you on 1. to an extent, but think that the amount of work that we can get done in a hard pause is probably fairly limited. This is mostly an intuition based on the fact that folks have been working on the theoretical questions for decades and that many of the remaining non-technical questions rely on people and governments taking AGI very seriously, something that is not the case right now and I believe is unlikely to become the case under a pause.
For point 2, I need to read more about feasibility of human intelligence amplifications. My first reaction though is that either we are very close to RSI/AGI/ASI in which case I think a global pause does not slow progress enough (due to enforcement issues) that long term plans are not good enough or we are not very close to RSI/AGI/ASI in which case we should hold off on a hard pause because the case for it will be stronger in the future. I think “giving everyone more time to understand the problem better and figure out ways to continue stopping” is a pretty strong argument, although I do worry that a stop would lower the salience of the issue.
people and governments taking AGI very seriously, something that is not the case right now and I believe is unlikely to become the case under a pause.
By a “pause”, the main thing I mean is “an international treaty to stop AGI globally”. I assume you’d think that’s unlikely because it’s unlikely that people + gvts would take it seriously enough. I don’t want to make a strong claim about it being likely feasible, and presumably even if it’s doable it would be a huge amount of hard work. But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
we are very close to RSI/AGI/ASI in which case I think a global pause does not slow progress enough (due to enforcement issues)
I mean, I agree that it’s kinda hard, but if AGI is very close, it probably involves lots of compute, right? Big piles of compute seem plausibly regulable. That doesn’t seem like an infeasible enforcement issue.
we are not very close to RSI/AGI/ASI in which case we should hold off on a hard pause because the case for it will be stronger in the future. …. although I do worry that a stop would lower the salience of the issue.
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
I don’t think you can talk about feasibility in a vacuum. Is it feasible that Congress pass a constitutional amendment removing the electoral college that is ratified by the states? From a technical sense, of course. From my perspective as someone who wants that to happen, not really.
Let me try to lay out my overall position.
I think right now you can break players into three categories
If all players are in group 1 (and knew the others were in group 1) a global pause is trivial. If all players are in groups 1 and 2 a global pause is a technical question: Can you create a treaty/system strong enough that secret defection is not possible? Players in group 3 have no interest in a pause and would have to be brought to the table with other incentives.
It’s very hard to know who is in which camp. I’d guess lab employees are largely split between 1, 2, and 3. I think the companies themselves are probably in groups 2 or 3.I think the U.S. government is in group 3 (caveat that there are lots of competing voices here and I have a very hard time modelling Trump). I have no idea where China or other world powers are.
The groups are more of a spectrum than binary. Someone in group 2 who thinks the status quo is very negative will be willing to spend more resources on a pause than someone who more loosely prefers a pause.
So I think the question of feasibility is how feasible is it for people in groups 1 and 2 to get the U.S. government into group 2 and strongly enough in group 2 that it is willing to expend lots of resources on a treaty (and I think since there are other players in group 3 you would need lots of resources). I don’t have a rock solid case that this is infeasible (I don’t think you ever really could), but I think it likely is at this time. EDIT: Adding in a little more here because I saw your conversations with others and think this may be our central disagreement. I think many people overrate how politically salient AI is. Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep. When you look at Gallup polling of the most important issues facing our country, AI doesn’t show up. I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished. I think if there are anti-AI legislation that is electorally valuable it will probably be surface level stuff that doesn’t move the needle on AI risk (e.g. data center water regulations). Now voters are not an immovable force, but I think it will be hard to move them quickly and may take some other shock to the system (impactful warning shot, widespread job loss, etc.)
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
As for why we might want to wait for a pause if AGI is further away, If we could have a strong pause now, I’d definitely agree with you. However, I think if you were to push the world into a weakly enforced pause it may be harder to get a strong pause down the line. In the status quo I hope we can slow down capabilities work, build up chip control systems and other monitoring options and as AI gets more powerful (hopefully before we all die) people will move towards a real pause. My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.
FWIW if I were a world dictator I would implement a global pause followed by heavily regulated and monitored technical alignment research. I also weakly hold the belief that people who think we are in an emergency situation should state that clearly and strongly, but have found some arguments for being more strategic compelling (e.g. Holden Karnofsky on the 80,000 hours podcast).
I think many people overrate how politically salient AI is.
I would say that much of my intuition that a pause (yes, driven by various national governments, and public desire) is plausibly doable comes from “the trajectory” of sentiments, rather than the total amount. I agree as a fraction of total political discourse it’s quite small.
Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep
This vaguely matches my impression for the most part, in the sense that if you look at discussion on AI in general, much of it may be negative but most of that isn’t people who deeply worry + care about x-risk. But to refine the picture, it’s mile wide, inch deep, but getting full of holes: it seems like there’s an ever growing number of people, including higher ups both in AI and also in government, who seem to take x-risk seriously, largely in words but also in more meaningful ways like drafting bills and stuff like that.
I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished.
Pardon my stupid question, but what goes wrong concretely? If you ban AGI, but let people keep running existing LLMs (say), does this really cause big and legible enough economic damage that voters would actually move? I mean, I’d think there’s lots of economically damaging things that don’t get credit-assigned into much actual vote shifts.
My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.
I may bow out, and feel free to take the maybe last, but I’ll just note that this still doesn’t make sense to me basically at all. Like, yes, there’s kinda-plausible scenarios where a global pause surprisingly ends up worse. But it would still be surprising, right? Like, there’s probably less human cloning right now, compared to the counterfactual where it wasn’t banned, right?? Someone could go underground with it, but that’s really hard and takes work!
I agree that slightly/somewhat unprecedentedly much access (official or espionage) might have to be somehow granted + enforced for treaty implementation. Maybe this is a strong defeater, I just don’t see it. Like, I agree it seems potentially kinda hard or quite hard in some scenarios. But this “global ban is actually worse than hugely resources companies going full tilt or even slightly less full tilt because of a “slowdown”″ seems galaxy-brained and false.
Is there a centralized explainer / synthesis / argument / whitepaper somewhere arguing a strong claim like “it’s quite likely infeasible to have a global strong slowdown / stop on AGI-ward research”?
Surely you could give an elaborate answer yourself.
I worry that this style of discussion is adversarial, and leads to conflict and polarization rather than advancing understanding.
Science is usually adversarial or advocacy based. It’s also usually very slow to arrive at consensus and thereby drive policy cleanly. We need to do better than traditional science.
If the outside perspective is just “experts disagree and argue,” decision-makers will be free to pick whatever perspective matches their Motivated reasoning, and confirmation bias.
Taking a collaborative and problem-solving approach may produce consensus more rapidly. It’s worth a shot. Arguing and pointing fingers is known to be slow and cause disagreement among experts.
And just to be clear, I think the fields’s consensus should be that alignment could be very tricky, so we should slow down progress if at all possible, and devote a lot more resources to alignment work.
Surely you could give an elaborate answer yourself.
I really genuinely couldn’t!
I can list reasons that it’s difficult, and I can list reasons that it’s quite difficult in the long run, assuming compute isn’t a big bottleneck. But I don’t believe that the assumption of long-run, non-compute-bottlenecked research being the main driver of AGI progress is an assumption held by Evan or by Anthropic. I could be incorrect about this, happy to be corrected.
I don’t know it to be infeasible to globally prevent AGI research that involves significant resources. Do you know this? I’m genuinely asking. I’m not aware of any serious case laid out for this. That’s my ignorance, someone could just link such to me! (I did run an AI search for such, without relevant results.)
Like, in my ignorance, as far as I’ve personally seen / tracked, Anthropic seems be almost entirely, though not entirely, messaging on the presumption that a global stop is infeasible; and they use that presumption as justification for leading frontier AGI research including RSI. Is this presumption seriously defended anywhere?
I see. I thought it was pretty clear why requirements to do more alignment work would be a lot easier and more attainable ask than a pause.
To answer your question: I don’t know where anyone has tried to carefully make the case that a pause is impossible. It’s more that there’s an assumption that it’s quite hard, so the burden of proof falls more on those calling for one to propose a plan that could work. I don’t think a pause is impossible, although I think it’s unlikely to get one right now soon.
So I think the assumption is that it would be easier to get requirements for alignment work as a requirement of US labs. With that, they could still stay ahead of Chinese development. That lowers the political bar substantially.
I personally hope that the government will take AGI and therefore misalignment risks more seriously as more people in govbernment see superhuman AI with their own eyes. I hope this will prompt them to slow down US progress, and try to negotiate cooperation with China. I”ve written about this in Whether governments will control AGI is important and neglected and have a mostly-finished draft followup with the working title “The goverment will assert control of AGI”.
I also think it’s not obvious that China is a less cautious actor. There have been recent good posts questioning this; China won’t win the AI race but would it be much worse if it did? and others have previously, including me, have pointed out that Chinese leadership seems much more cautious by nature. And others have pointed out that, in the case of successful intent alignment, having the Chinese government taking over the future really probably wouldn’t be much worse in expectation than having the US government take it over. But that’s a tougher sell for political purposes. We probably can’t get through a US government commitment to unilaterally pausing, even once they do see the danger of misalignment more clearly. But I suspect the Chinese would be pretty eager for any deal other than “we’ll race as fast as we can and maybe get everyone killed and pretty much take over the world if we get aligned ASI first”.
On Anthropic, even though that’s not the focus of your question: I don’t think Anthropic have sounded like they’ve been claiming that pause is impossible, just there’s not one on offer. Them pausing won’t make others pause. This does raise the question of why they’re not mentioning that they’d pause if others did. I believe they’re now on record stating they would (although of course this isnt’ a firm commitment). In their recent post When AI builds itself on RSI and the implications, they explicitly say that it would be wise to slow down the rush to ASI if “less cautious” actors don’t keep going full speed. And they say the Anthropic Institute will be studying how to do that. This is somewhat beside the point, except that it is a bit more explicit statement of their position.
I thought it was pretty clear why requirements to do more alignment work would be a lot easier and more attainable ask than a pause.
Well, Evan wrote:
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable
It seems more attainable, sure, but also way way less useful, to the point where I don’t understand how it’s better than even something as sketchy as a napkin with “Advocate for a global stop on AGI research” written on it. Do you disagree?
We probably can’t get through a US government commitment to unilaterally pausing
I’m not especially advocating for that, though it might be good, IDK. I’m advocating for Anthropic to advocate for a global agreement.
I don’t know where anyone has tried to carefully make the case that a pause is impossible. It’s more that there’s an assumption that it’s quite hard, so the burden of proof falls more on those calling for one to propose a plan that could work. I don’t think a pause is impossible, although I think it’s unlikely to get one right now soon.
Do you agree that this assumption is a founding pillar of Anthropic’s strategy, at least as they present it? If this assumption is justifying doing frontier AGI research, should it be, like, argued somewhere? Don’t you think it’s weird that Anthropic as a whole big entity seems to be acting on this assumption, without a public argument for it? Do you think there’s no good argument for it, or do you think that it’s all in private somehow?
I do not work at Anthropic or any other developer. I’m just extending their thinking the same benefit of the doubt I extend to everyone.
You quote Evan saying this proposal is better than other slowdown proposals for reasons other than being attainable. You originally asked “why not a pause” and the obvious answer, which I think you could supply yourself, was “this is far more attainable”. This is entirely consistent with (although not explicitly mentioned in) Evan’s framing.
Just to add a bit in case you genuinely haven’t thought about why achieving a pause is so hard right now: if you got the US labs to all pause somehow, this would not likely get the Chinese labs to pause, or prevent new US labs from springing up to rush ahead. If the argument were strong enough, then sure it would get everyone to pause.
The fundamental problem, IMO, is that the arguments aren’t strong enough. Most of the developers have misalignment x-risks at maybe something like ten percent. You can argue that they’re all wrong, because they have motivated reasoning from the incentives. I think this is correct, but it adds a second layer of complexity to the argument.
Another factor that’s not obvious: I think the world would happily rush ahead toward AGI if extinction risks were only about 10%. If we get immortality and solve all of our problems, most people are not utilitarians and would trade that much risk of disaster for their own gain—and skipping the hassle of trying to stop it.
Just to reiterate, I think that we should pause development. You asked why that’s hard, so I’m answering—even though I still think your questions may include a degree of socratic trolling I think is ultimately unhelpful. I’ve gotten some benefit from taking that bait.
This discussion has pushed me to consider options a little more. I think there’s a chance the US government could unilaterally push Chinese labs to do a similar amount of alignment research as outlined in Evan’s original proposal. I think they could potentially get through a law prohibiting US businesses from using models developed without alignment research. But that’s on the original proposal, not the pause you were asking about.
Ah ok here’s something that’s implicit, that I can now make explicit: IF you’re taking big actions (e.g. Anthropic doing frontier AGI research) AND your justifications for doing that rely on a quite unclear claim X, THEN you should have reasoned out X pretty well and defend it publicly OR ELSE you probably are deceiving yourself and/or others about why you’re doing what you’re doing. (There’s plenty of exceptions, but still.)
Sorry, I wasn’t fully explicit. When I said “Advocate for a global stop on AGI research”, I meant for us (broadly, including Evan, say) to advocate to world governments that they should institute a global stop, not that anyone’s supposed to persuade labs to stop. (I mean, separately I’d like to push on many points of intervention, including persuading AGI researchers to stop it.)
I still think your questions may include a degree of socratic trolling I think is ultimately unhelpful
I’m really not. Not sure why you think that. I can try to lay more things out, though I think I’ve indicated some / most of this? Anyway:
I think it’s plausible that a global stop could be coordinated. I’m a layman, so I can’t, like, argue for this super well or coherently.
I think that it’s bad for labs to be doing AGI research.
Anthropic seems to justify pushing the frontier by saying a global pause is probably not doable.
I have literally never heard a serious coherent case laid out that this is not doable. It’s just something that it’s embarrassing if you don’t understand. But that’s garbage reasoning.
If I had to guess, I would guess that most of them (non-leadership Anthropic employees) broadly are just mindfucked somehow. However, I see plenty of bits of apparent good faith and good intentions, I know some of the people there and think they are smart and have or had good intentions, etc. Also, generally it seems rude to just assume there is no such case with checking. So, I have been trying to check by asking some Anthropic employees, and a bit of searching, and asking on LW. So far, no one has said “oh yeah here’s a link”, but people continue to argue or otherwise act as though it’s obvious. You tell me what I’m supposed to make of this! Maybe this is what comes off as trolling? Maybe we really should unban Said!
Gotcha. That all makes sense. I agree on almost every point. I definitely think we should be advocating for a global stop on AGI research, as well as slowdowns, as well as working on alignment research as fast as we possibly can, while trying to avoid advancing capabilities faster than alignment. These aren’t at all mutually exclusive. Everyone who sees the problem should be doing all of them IMO. I haven’t seen strong arguments against putting effort into all of these at once.
I think it IS obvious why it’s hard. Any international agreement is hard, and even people who think about p(doom) think it’s on the order of 10% and are mostly basically okay pushing ahead on those odds. This brings us to the “mindfucked somehow”. I think that’s probably basically right. I’ve laid out what I think are the relevant mechanisms, in a great deal of detail and with a great deal of empirical backing, in the post I referenced but didn’t explain: Motivated reasoning, confirmation bias, and AI risk theory. I wouldn’t use such a strong term because I don’t think it’s as strong as that implies—and calling people mindfucked is a great way to even further motivate them against you and everything associated with you.
Which brings us to the charge of socratic trolling. I certainly believe you when you say that’s not what you’re trying to do. But your intention has only a weak causal relationship to whether the effect is net-helpful or not. I had Said in mind when that term popped into my head. I followed that drama with great interest, and have very mixed feelings about it, but I think there’s a legitimate concern there. I think three years was way too long, and in the first go-round didn’t agree with Duncan Sabien’s calls for his banning (despite his amusingly biting the bullet and arguing it was good that Socrates was banished/killed for similar behavior). But I found Habryka’s arguments in Banning Said Achmiz (and broader thoughts on moderation) convincing in the face of Zach MD’s recent objections in Comment on “Banning Said Achmiz”. I’m not sure the time invested in all of those massive posts was worth it, but I feel I did exit with a little wisdom on the dynamics of what I’m going to keep calling socratic trolling, because it mixes those virtues and costs.
I believe you’re being completely sincere, and that you actually had very good reasons for asking that question, after this exchange. I think you’re correct that this assumption deserves to be questioned and debated.
Framing it as you did without context looks somewhat combative to my eye, and also seems like an isolated demand for rigor. So I stand by my initial assessment of “concerning and probably more harm than good”. I think it’s pretty easy to mitigate the downsides while keeping the upsides. Showing proof of effort by laying out the reasons you’re asking make it clear that you’re not trolling. And deliberately framing the question in a friendly way avoids the look of it being combative or adversarial, and thereby sparking more argument and “mindfucking” polarization (on both sides, obviously, since rationalists unfortunately still have feelings and motivations and halo/horns effects).
Anyway, this has been a useful exchange and I appreciate it. I think we’re all amateurs at what it takes to get a global treaty on pausing AI research; experts on international treaties are not at all experts on the argument that would drive it forward, and vice-versa.
Anthropic did explicitly say in their recent piece on RSI that they would start investigating how to get a pause or slowdown. It seems like other concerned parties like you and I should also chip in a little research here and there, even if it’s not our domain of direct expertise.
Framing it as you did without context looks somewhat combative to my eye,
I usually have a pretty bad intuitive reaction to people telling me that my writing is “too combative”, but I don’t know why fully, and in a small fraction of the cases I end up agreeing with them later, so I won’t respond further. I do want to say what I just said, though; not sure why, but maybe, for example, I feel that it’s somehow unfair, though since I can’t explain how this is a very unreliable sense.
and also seems like an isolated demand for rigor
I think it’s a really important and central supposition, and apparently has not been rigorously defended anywhere! This is 100% definitely not what the concept of “isolated demand for rigor” is for! What other such things am I allegedly not demanding rigor for, that I ought to be? Or do you disagree / am I confused somehow?
So I stand by my initial assessment of “concerning and probably more harm than good”.
Ok. (If you wanted to update me personally, you haven’t done that on this point.)
Showing proof of effort by laying out the reasons you’re asking make it clear that you’re not trolling.
Until this point, I have been genuinely unsure if there’s simply a report / blog post / something that someone might just link me to, explaining the case!! But yes, I think you’re right, I now agree it would be better for a comment like my first to come with a few sentences explaining that it seems like there isn’t such a case, that a global stop seems plausibly feasible, that Anthropic seems super far from appropriately supporting that, and that they should.
Fair. I’m aware that people react negatively to being tone policed. Somewhat ironically, this can motivate people against the concept of motivated reasoning effects I think are pretty crucial to our understanding and therefore our survival. So I’ve avoided doing that for the most part, and regretted doing it. I pressed on because you didn’t immediately react badly. I’m still not sure if bringing this up is harmful or helpful.
But I have heard people from the developer side of the fence say that they find LW a hostile environment and have trouble engaging here even though they feel they should. And the tone of discussions here certainly look like tribal dynamics and polarization are happening. So I think this is an important topic, although I’m not at all sure I’m raising it the right way. That big MR post was my due diligence on making sure I know what I’m talking about; now I need to decide whether and how to push the issue more forcefully.
I expect I haven’t changed your mind because you haven’t read my careful research and arguments. I wouldn’t expect you to change your mind without evidence. Of course the evidence I present in that post isn’t airtight, but to me it looks awfully likely that being at least somewhat deliberately warm/nice is worthwhile if you want to win people over to your side. That seems particularly likely if there’s anything debatable at all. If the subject is really cut-and-dried I think you can push harder and succeed, but you still slow rate of progress if your approach is setting off enemy-warning signals in the audience’s brains.
Anyway, thanks for engaging that far. This is an important topic to me, and I haven’t really engaged on it here before. So I appreciate it.
On your other point: I am also unsure if there’s a report or blog post that more thoroughly lays out the case for pause being impossible. I doubt there’s a particularly good one.
The inverse of such a report is a careful arguement for how it is possible. I’ve laid out bits and pieces of a case for how slowdown is quite plausible, even possibly as a default on the current trajectory, but not in one coherent place or for a full pause. But the arguments can be extended for the possiblity of a pause.
I won’t dive into that further right now, but I think it is a worthy collaborative project for LWers.
But I have heard people from the developer side of the fence say that they find LW a hostile environment and have trouble engaging here even though they feel they should. And the tone of discussions here certainly look like tribal dynamics and polarization are happening.
This makes sense. I will note however that when I (one time) asked an Anthropic employee about inviting someone over to their offices to explain / argue more in depth some crucial point (I forget; I think alignment difficulty), they said something like “last one or two times we tried that, the guest was dismissive”. So like, it looks a whole lot more like the crux is self-insulation, even if there is also undue hostility on LW. But, that is N=1. (I have other Ns that look like self-insulation, though of course that’s almost inherently ambiguous and my total N is small.)
(I do think in past I’ve at least watched from the sidelines, or even slightly participated in, arguably-undue dogpiley polite arguing, if not hostility.)
I won’t dive into that further right now, but I think it is a worthy collaborative project for LWers.
Interesting. Both of those posts have the form of “what would an agreement say” which I think is totally missing the hard part. So I think that points at the answer to your original question, and why others regard it as obvious and you do not.
The answer is “because there’s no political will”. And so the question isn’t what would an agreement say, it’s where would the political will come from.
My answer is that the political will will come from AI progress, particularly from visible job loss and from human-seeming AI systems, which will trip pattern-matching to strange humans, which we intuitively regard as quite dangerous. Xenophobia exists for a reason; strange humans have been among our biggest dangers since the start of evolution. I’ve written about this in A country of alien idiots in a datacenter: AI progress and public alarm and bits and pieces elsewhere.
On the positive side, this answer is that the political will will come, which shifts the question back to having an agreement or treaty ready to offer.
The problem with this answer is that the will might come too late. By the time systems are visibly taking jobs and acting agenticly and competently, we might already have a takeover-capable system in development, and it will be too late for anything as slow as international agreements. There still might be time for an executive order and informal power grabs in the face of adequate public (and politician) freakout. And that might be substantially useful, since only China is near the US, and they’d probably take a much more cautious approach to AGI and alignment (see China won’t win the AI race but would it be much worse if it did? and similar). I’ve written about this in Whether governments will control AGI is important and neglected and I now think the answer is just clearly yes, but maybe not in time.
That’s a slowdown not a pause, but it could perhaps be expanded to a pause if the discussion can move quickly between the US and China. I think this is possible. We’re really not enemies, just competitors. And our researchers are quite well-disposed toward each other.
Anyway, I think the major crux between us and the rest of the world is alignment risk. The average belief even among those that acknowledge the risk (which tbf is now pretty much anyone thinking about AGI) is maybe ten percent or lower. That’s enough to make some people want to pause, but not I think most of them, and not enough to make it a high priority.
So I think the clearest path toward pause (or slowdown) is clearer arguments about alignment risk.
Here I think overclaiming on the technical arguments has done grevious harm to the cause. Yudkowsky’s claim that misalignment is 99%+ likely has drawn much irritation and ire and attention. People, even sophisticated people, routinely argue “alignment is possible” instead of arguing about how possible. I think they’re quite correct that Yudkowsky’s technical argument is full of holes, but quite wrong in the implied leap from there to ~10% risk. Alignment may be quite achievable and still quite difficult to achieve on the current rushed path.
I think arguments for human incompetence on first tries and under pressure are a much better bet. To his credit, EY has shifted hard in this direction, and so have the handful of others making technical arguments for alignment difficulty. My arguments center on model uncertainty: nobody knows how hard alignment is; estimates from people with real time-on-task range from very low to very high; therefore the wisest assumption is that it could be extremely difficult and we are foolish to press ahead with so much unknown.
Here I think we could do vastly better. Optimists reason that current systems seem pretty aligned, so we’re probably on track to align more powerful systems. Pessimists argue that this isn’t useful evidence at all. Identifying cruxes and improving models of likely first AGI seems quite achievable, so that’s what I’m primarily working toward and asking others to engage in.
WRT the Anthropic office visits: This has the general form of “it’s their fault not ours” which is suspicious. In most disagreements, both parties blame the other. And even if it is totally their fault, I’d rather survive than assign blame. Usually the way forward in resolving interpersonal issues is “sorry about that, let’s try again” and then be nicer.
This is when you’re trying to reach mutual agreement with someone, not when you’re trying to negotiate a deal and have some leverage. Discussions about beliefs only resemble negotiations when the evidence is overwhelming. And on the dangers of alignment, it’s just unfortunately not.
Both of those posts have the form of “what would an agreement say” which I think is totally missing the hard part. So I think that points at the answer to your original question, and why others regard it as obvious and you do not.
You might be overinferring what I think these blog posts indicate? I’m just gesturing that I agree that the overall project of figuring out how the whole thing might be feasible is a worthy project.
The answer is “because there’s no political will”.
I know that this is a thing people say, and I agree there isn’t already automatically political will pre-gathered. But if the implication is that it would be an infeasible task to create and gather the political will for a global stop, that implication is one I strongly question! And so far I hear lots of signs pointing in the opposite direction, and grateful to the people working on that. I just wish that Anthropic would support those efforts.
WRT the Anthropic office visits: This has the general form of “it’s their fault not ours” which is suspicious.
Not blaming, describing. Can’t survive without describing.
(Anyway, just FYI, your time might be somewhat wasted if you want to get me on board with a particular approach / stance, because I’m much more commenting from the sidelines rather than an active participant; I’m focusing on other things, while others are actually working on communicating with the public and political leaders and so on.)
I’m also primarily occupied with other things. I’m spending some time on communication strategy and the logic of how opinion and policy could change, because it seems like it could be critically important, and not enough people seem to be thinking about it. As you note.
Oh—I left out one important consideration in my theory of how people get “mindfucked” in Motivated reasoning, etc: social dynamics. When I dug into the logic and the science deeply, I wound up thinking that social networks of who you respect and who they respect, and the resultant “double-counting”—actually many-counting—are probably even stronger effects than the strong individual effects I could put at least approximate empirically supported numbers to. (I actually think you can measuer network effects, I just had spent way too long on the project to dive into another area of research I wasn’t already familiar with). Anyway: mutual respect dynamics are probably strong, and they’re probably influenced strongly by motivated reasoning/horns&halo effects.
This is another reason to make sure to avoid polarization and an us-vs-them attitude. It makes everyone more mindfucked, in multiple and empirically strong ways.
Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
Something I’ve thought before is that it seems like most people are rolling their own conclusion about the political feasibility of pausing. They think about it for five minutes or less, and then they’re done; they decide whether building a US pause coalition sounds reasonable, or whether China could be cooperated with, mostly on priors. There’s no Rootclaim for the politics of an AI pause. No one org owns a pipeline for doing this research, not even for the narrow version of message testing. There’s just PauseAI, ControlAI, and StopAI doing their own scattered advocacy efforts live, with close to no support. Vastly influential decisions that caused hundreds of millions to flow to technical safety, and at most a few million to advocacy, were made mostly on vibes. No one even did A/B testing or focus groups for pause messaging until last year!
Mind that I don’t mean the technical implementation or effects of a pause, like with what MIRI does or what the 2023 AI Pause Debate did. I mean whether it seems politically achievable, whether it’s even in the Overton window, or if the window can be moved there. We saw with how PEPFAR was founded that sometimes a political miracle can just happen if there’s the will for it.
I think the question of “is a pause politically feasible” should have an adversarial collaboration done on it. That’s one of the best methods of truth-finding I know of, and it’s sad we don’t do more of it.
I suspect this may be the case, but I’d like to know how people think about their beliefs about this question, but the way Anthropic employees (n=3ish) act when I try to ask about it is confusing. I wish they would just say “I’m relying on the judgement of so-and-so” or something like that, as it would be much clearer. (Cf. https://www.lesswrong.com/posts/zLG2DnJw6oEZqAgaE/tools-for-deferring-gracefully )
I don’t know it to be infeasible to globally prevent AGI research that involves significant resources. Do you know this?
By “infeasible”, do you mean strictly “technically infeasible / infeasible even if there’s the will to make it happen”, or something that also includes “tractable / politically palatable”?
I like this! Here’s a specific concrete version of it just to have some numbers on the table:
Companies are legally required to allocate their compute as follows: 1. 60% on serving customers. 2. 15% on internal safety research that’s fully transparent to the public. 3. 10% on external safety research that’s fully transparent to the public & has access to the latest internal models. 4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)
Suppose this were a US law, that applies to all the major AI companies but not to e.g. Chinese companies or Mistral for example.
15% is less than the roughly 50% they seem to be spending now, so overall the pace of AI progress would slow down to maybe something like half the speed it’s going now. But the amount of compute devoted to capabilities progress would still be higher at each US company than any Chinese company for example, so US companies would stay in the lead.
Meanwhile the amount of internal compute going to safety research would double or quadruple, and then the effect size of a ton of external researchers being able to look at the work, critique it, run their own replications and ablations, run their own experiments, etc. would be way way bigger on top of that already big positive effect.
How would it be enforced? Like you said, whistleblowers. If we were trying to ban AI capabilities R&D entirely, that would be super difficult, but if we are just keeping it to 15% of the compute, then cheaters can’t really prosper. It’s really hard to hide 10% of your compute being spent on something it’s not supposed to be spent on; maaayybe you can hide 1% but that wouldn’t be enough to make much difference & so wouldn’t be worth the risk.
This would also prevent a sort of race to the bottom where companies use more and more of their compute on R&D instead of serving customers; instead we’d have the process of AI deployment/diffusion continuing to happen which I think is broadly pretty good for the world.
I don’t feel strongly about the specific numbers above obviously but yeah this seems promising.
Here’s a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):
Enforce that labs must spend at least X% of their compute on:
Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won’t be as relevant to the labs’ specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research the lab tries to do via this mechanism much less likely to advantage them differentially (though of course they could still just end up doing bad safety research if they pick bad external safety orgs).
Internal safety research that the lab makes fully public. Again I think this substantially disincentivizes capabilities research, since any capabilities research that is fully public will be much less able to differentially advantage that lab specifically, as others can just copy it (though again of course the labs could still just do bad public safety research).
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible—and it also differentially advantages alignment research by choosing conditions (external or fully public research) that make it much harder to do capabilities under those conditions but still pretty easy to do alignment. If you just say that compute has to go to some sort of nebulous “safety” purpose, that could easily be gamed, but I think the conditions above do a pretty good job of operationalizing criteria that are fairly hard to game. And even if you do get some gaming, on the margin you are still making capabilities research more costly, and thus buying yourself some additional slowdown, as well as making alignment research less costly, and thus buying yourself some additional alignment research.
(A possible second-best proposal if this is unworkable for some reason is just to enforce X% of compute go to external inference of some variety, which is more concrete and still gets you lots of slowdown, but less differentially advantages alignment research.)
What’s wrong with just stopping research, period?
If you think that alignment is somewhat tractable and that any sort of pause won’t be able to constrain 100% of labs then you are just buying yourself a finite amount of time. You then have to actually do safety research during that time.
Stopping all research would be much more difficulty politically in the U.S.
As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.
If you are very confident that alignment is very very hard and think that any warning signs found during the additional safety research will definitely be ignored, I agree this doesn’t do much.
Why do you think this? (Where “all” is not 100% literal, but close to literal.) (What you say is plausible, but I don’t know it.)
Let’s assume it’s happening globally; or else, I wonder why that’s viewed as clearly infeasible.
I think nuclear non-proliferation is a good benchmark here. The largest powers really cared about non-proliferation and did a good, but not perfect job (we’ve had four nations become nuclear since the NPT). I think many people overrate how difficult it would be to monitor/constrain AI research because of how specialized the GPU supply chain is, but I still think it’s harder to monitor than nuclear weapon development (Probably, I haven’t read up on this a lot).
Non-proliferation is not a perfect analogy for different reasons, but I think those reasons push in different directions and would need a strong argument on why we would do better with AI.
Evan’s OP said:
I guess you’re taking this to apply specifically to the US, but not other places? Or is the idea something like:
Does this argument apply to nukes? Why / why not? How could espionage play a role in global enforcement?
I’m a bit skeptical (but quite unsure) that it’s very feasible to carry out significant AI research efforts without anyone knowing. How do you recruit scientists without anyone noticing, for example? E.g. if you approached a bunch of serious scientists to develop a serious bioweapon, I think / would hope (??) that you’d quickly get reported to something??
[Discourse note: I personally don’t have informed or strong opinions about this feasibility, short term or long term; what I’m confused about is the discourse that seems to assume a strong position that it’s not feasible, and I’m trying to check whether there’s a model behind that, or I’m missing something else, or what.]
First a clarification: In your initial post I took “stopping research” to mean stopping all AI research without a safety carve out. If that’s not the case please correct me.
Let’s say any policy intervention has a capabilities progress multiplier (c). So if at our current rate we are x years from developing powerful AI (defined as having a better than 50% chance of being able to take over the world for it’s creator if aligned or for itself if not aligned) then it would take x/c years under that policy. Let’s also think of an safety progress multiplier (s) so that in y/s years we will be confident we can safely align powerful AI where y is how far we are at the current rate.
We want a policy that either makes x/c so large we don’t care about it or makes x/c>y/s. I need to do more research, but my initial though is that making y/c so large we don’t care about it is very hard. If making y/c so large we don’t care about it seems plausible to you, we can get into the details more. I have some intuitions here, but need to do more research.
Assuming we are not on track to solve alignment (which I think we would both agree is likely true) then we know x<y. Thus any policy intervention needs to have s>c. I think that is true of Evan’s policy, but not true of a global moratorium (because defectors are likely to differentially prioritize capabilities over research).
You can do alignment research without doing current-flavored capabilities research. For example, no one has the concepts needed to understand values, which would probably be needed to do alignment; this can, and probably must, be investigated conceptually.
An intervention has many effects and differentially affects many processes, not just capabilities and alignment. Cf. https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#An_ontology_of_effects_of_interventions_on_world_processes For example, a stop gives everyone more time to understand the problem better and figure out ways to continue stopping. It also gives more time for human intelligence amplification, and other ways that humanity can make itself more able to survive the possibility of AGI.
Before we get to these points I want to rehilight and briefly expand on the other two points in my original comments.
Stopping all research would be much more difficulty politically in the U.S.
This would erase trillions of dollars of value, potentially 10s of trillions, something the U.S. government has a lot of incentives not to do.
As Kokotajlo points out, this could feasibly be done just in the U.S. without the U.S. losing their lead.
To get anything close to a real global pause the U.S. or others would have to commit (and probably take) military action to enforce it, again something the U.S. government has a lot of incentives not to do.
To address your points. I agree with you on 1. to an extent, but think that the amount of work that we can get done in a hard pause is probably fairly limited. This is mostly an intuition based on the fact that folks have been working on the theoretical questions for decades and that many of the remaining non-technical questions rely on people and governments taking AGI very seriously, something that is not the case right now and I believe is unlikely to become the case under a pause.
For point 2, I need to read more about feasibility of human intelligence amplifications. My first reaction though is that either we are very close to RSI/AGI/ASI in which case I think a global pause does not slow progress enough (due to enforcement issues) that long term plans are not good enough or we are not very close to RSI/AGI/ASI in which case we should hold off on a hard pause because the case for it will be stronger in the future. I think “giving everyone more time to understand the problem better and figure out ways to continue stopping” is a pretty strong argument, although I do worry that a stop would lower the salience of the issue.
By a “pause”, the main thing I mean is “an international treaty to stop AGI globally”. I assume you’d think that’s unlikely because it’s unlikely that people + gvts would take it seriously enough. I don’t want to make a strong claim about it being likely feasible, and presumably even if it’s doable it would be a huge amount of hard work. But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
I mean, I agree that it’s kinda hard, but if AGI is very close, it probably involves lots of compute, right? Big piles of compute seem plausibly regulable. That doesn’t seem like an infeasible enforcement issue.
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
I don’t think you can talk about feasibility in a vacuum. Is it feasible that Congress pass a constitutional amendment removing the electoral college that is ratified by the states? From a technical sense, of course. From my perspective as someone who wants that to happen, not really.
Let me try to lay out my overall position.
I think right now you can break players into three categories
Those who model the situation as a stag hunt.
Those that model the situation as a prisoner’s dilemma.
Those that model the situation as a deadlock.
If all players are in group 1 (and knew the others were in group 1) a global pause is trivial. If all players are in groups 1 and 2 a global pause is a technical question: Can you create a treaty/system strong enough that secret defection is not possible? Players in group 3 have no interest in a pause and would have to be brought to the table with other incentives.
It’s very hard to know who is in which camp. I’d guess lab employees are largely split between 1, 2, and 3. I think the companies themselves are probably in groups 2 or 3.I think the U.S. government is in group 3 (caveat that there are lots of competing voices here and I have a very hard time modelling Trump). I have no idea where China or other world powers are.
The groups are more of a spectrum than binary. Someone in group 2 who thinks the status quo is very negative will be willing to spend more resources on a pause than someone who more loosely prefers a pause.
So I think the question of feasibility is how feasible is it for people in groups 1 and 2 to get the U.S. government into group 2 and strongly enough in group 2 that it is willing to expend lots of resources on a treaty (and I think since there are other players in group 3 you would need lots of resources). I don’t have a rock solid case that this is infeasible (I don’t think you ever really could), but I think it likely is at this time. EDIT: Adding in a little more here because I saw your conversations with others and think this may be our central disagreement. I think many people overrate how politically salient AI is. Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep. When you look at Gallup polling of the most important issues facing our country, AI doesn’t show up. I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished. I think if there are anti-AI legislation that is electorally valuable it will probably be surface level stuff that doesn’t move the needle on AI risk (e.g. data center water regulations). Now voters are not an immovable force, but I think it will be hard to move them quickly and may take some other shock to the system (impactful warning shot, widespread job loss, etc.)
As for why we might want to wait for a pause if AGI is further away, If we could have a strong pause now, I’d definitely agree with you. However, I think if you were to push the world into a weakly enforced pause it may be harder to get a strong pause down the line. In the status quo I hope we can slow down capabilities work, build up chip control systems and other monitoring options and as AI gets more powerful (hopefully before we all die) people will move towards a real pause. My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.
FWIW if I were a world dictator I would implement a global pause followed by heavily regulated and monitored technical alignment research. I also weakly hold the belief that people who think we are in an emergency situation should state that clearly and strongly, but have found some arguments for being more strategic compelling (e.g. Holden Karnofsky on the 80,000 hours podcast).
I would say that much of my intuition that a pause (yes, driven by various national governments, and public desire) is plausibly doable comes from “the trajectory” of sentiments, rather than the total amount. I agree as a fraction of total political discourse it’s quite small.
This vaguely matches my impression for the most part, in the sense that if you look at discussion on AI in general, much of it may be negative but most of that isn’t people who deeply worry + care about x-risk. But to refine the picture, it’s mile wide, inch deep, but getting full of holes: it seems like there’s an ever growing number of people, including higher ups both in AI and also in government, who seem to take x-risk seriously, largely in words but also in more meaningful ways like drafting bills and stuff like that.
Pardon my stupid question, but what goes wrong concretely? If you ban AGI, but let people keep running existing LLMs (say), does this really cause big and legible enough economic damage that voters would actually move? I mean, I’d think there’s lots of economically damaging things that don’t get credit-assigned into much actual vote shifts.
I may bow out, and feel free to take the maybe last, but I’ll just note that this still doesn’t make sense to me basically at all. Like, yes, there’s kinda-plausible scenarios where a global pause surprisingly ends up worse. But it would still be surprising, right? Like, there’s probably less human cloning right now, compared to the counterfactual where it wasn’t banned, right?? Someone could go underground with it, but that’s really hard and takes work!
I agree that slightly/somewhat unprecedentedly much access (official or espionage) might have to be somehow granted + enforced for treaty implementation. Maybe this is a strong defeater, I just don’t see it. Like, I agree it seems potentially kinda hard or quite hard in some scenarios. But this “global ban is actually worse than hugely resources companies going full tilt or even slightly less full tilt because of a “slowdown”″ seems galaxy-brained and false.
I’d also point out that a treaty isn’t a thing that happens, and then whatever’s written in the treaty determines how well the treaty helps the situation. A treaty is a step in a broader process that can continue to adapt and develop to avoid the dangerous stuff happening. Cf. https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad
Is there a centralized explainer / synthesis / argument / whitepaper somewhere arguing a strong claim like “it’s quite likely infeasible to have a global strong slowdown / stop on AGI-ward research”?
Surely you could give an elaborate answer yourself.
I worry that this style of discussion is adversarial, and leads to conflict and polarization rather than advancing understanding.
Science is usually adversarial or advocacy based. It’s also usually very slow to arrive at consensus and thereby drive policy cleanly. We need to do better than traditional science.
If the outside perspective is just “experts disagree and argue,” decision-makers will be free to pick whatever perspective matches their Motivated reasoning, and confirmation bias.
Taking a collaborative and problem-solving approach may produce consensus more rapidly. It’s worth a shot. Arguing and pointing fingers is known to be slow and cause disagreement among experts.
And just to be clear, I think the fields’s consensus should be that alignment could be very tricky, so we should slow down progress if at all possible, and devote a lot more resources to alignment work.
I really genuinely couldn’t!
I can list reasons that it’s difficult, and I can list reasons that it’s quite difficult in the long run, assuming compute isn’t a big bottleneck. But I don’t believe that the assumption of long-run, non-compute-bottlenecked research being the main driver of AGI progress is an assumption held by Evan or by Anthropic. I could be incorrect about this, happy to be corrected.
I don’t know it to be infeasible to globally prevent AGI research that involves significant resources. Do you know this? I’m genuinely asking. I’m not aware of any serious case laid out for this. That’s my ignorance, someone could just link such to me! (I did run an AI search for such, without relevant results.)
Like, in my ignorance, as far as I’ve personally seen / tracked, Anthropic seems be almost entirely, though not entirely, messaging on the presumption that a global stop is infeasible; and they use that presumption as justification for leading frontier AGI research including RSI. Is this presumption seriously defended anywhere?
I see. I thought it was pretty clear why requirements to do more alignment work would be a lot easier and more attainable ask than a pause.
To answer your question: I don’t know where anyone has tried to carefully make the case that a pause is impossible. It’s more that there’s an assumption that it’s quite hard, so the burden of proof falls more on those calling for one to propose a plan that could work. I don’t think a pause is impossible, although I think it’s unlikely to get one right now soon.
So I think the assumption is that it would be easier to get requirements for alignment work as a requirement of US labs. With that, they could still stay ahead of Chinese development. That lowers the political bar substantially.
I personally hope that the government will take AGI and therefore misalignment risks more seriously as more people in govbernment see superhuman AI with their own eyes. I hope this will prompt them to slow down US progress, and try to negotiate cooperation with China. I”ve written about this in Whether governments will control AGI is important and neglected and have a mostly-finished draft followup with the working title “The goverment will assert control of AGI”.
I also think it’s not obvious that China is a less cautious actor. There have been recent good posts questioning this; China won’t win the AI race but would it be much worse if it did? and others have previously, including me, have pointed out that Chinese leadership seems much more cautious by nature. And others have pointed out that, in the case of successful intent alignment, having the Chinese government taking over the future really probably wouldn’t be much worse in expectation than having the US government take it over. But that’s a tougher sell for political purposes. We probably can’t get through a US government commitment to unilaterally pausing, even once they do see the danger of misalignment more clearly. But I suspect the Chinese would be pretty eager for any deal other than “we’ll race as fast as we can and maybe get everyone killed and pretty much take over the world if we get aligned ASI first”.
On Anthropic, even though that’s not the focus of your question: I don’t think Anthropic have sounded like they’ve been claiming that pause is impossible, just there’s not one on offer. Them pausing won’t make others pause. This does raise the question of why they’re not mentioning that they’d pause if others did. I believe they’re now on record stating they would (although of course this isnt’ a firm commitment). In their recent post When AI builds itself on RSI and the implications, they explicitly say that it would be wise to slow down the rush to ASI if “less cautious” actors don’t keep going full speed. And they say the Anthropic Institute will be studying how to do that. This is somewhat beside the point, except that it is a bit more explicit statement of their position.
Well, Evan wrote:
It seems more attainable, sure, but also way way less useful, to the point where I don’t understand how it’s better than even something as sketchy as a napkin with “Advocate for a global stop on AGI research” written on it. Do you disagree?
I’m not especially advocating for that, though it might be good, IDK. I’m advocating for Anthropic to advocate for a global agreement.
Do you agree that this assumption is a founding pillar of Anthropic’s strategy, at least as they present it? If this assumption is justifying doing frontier AGI research, should it be, like, argued somewhere? Don’t you think it’s weird that Anthropic as a whole big entity seems to be acting on this assumption, without a public argument for it? Do you think there’s no good argument for it, or do you think that it’s all in private somehow?
BTW, do you work at Anthropic?
I do not work at Anthropic or any other developer. I’m just extending their thinking the same benefit of the doubt I extend to everyone.
You quote Evan saying this proposal is better than other slowdown proposals for reasons other than being attainable. You originally asked “why not a pause” and the obvious answer, which I think you could supply yourself, was “this is far more attainable”. This is entirely consistent with (although not explicitly mentioned in) Evan’s framing.
Just to add a bit in case you genuinely haven’t thought about why achieving a pause is so hard right now: if you got the US labs to all pause somehow, this would not likely get the Chinese labs to pause, or prevent new US labs from springing up to rush ahead. If the argument were strong enough, then sure it would get everyone to pause.
The fundamental problem, IMO, is that the arguments aren’t strong enough. Most of the developers have misalignment x-risks at maybe something like ten percent. You can argue that they’re all wrong, because they have motivated reasoning from the incentives. I think this is correct, but it adds a second layer of complexity to the argument.
Another factor that’s not obvious: I think the world would happily rush ahead toward AGI if extinction risks were only about 10%. If we get immortality and solve all of our problems, most people are not utilitarians and would trade that much risk of disaster for their own gain—and skipping the hassle of trying to stop it.
Just to reiterate, I think that we should pause development. You asked why that’s hard, so I’m answering—even though I still think your questions may include a degree of socratic trolling I think is ultimately unhelpful. I’ve gotten some benefit from taking that bait.
This discussion has pushed me to consider options a little more. I think there’s a chance the US government could unilaterally push Chinese labs to do a similar amount of alignment research as outlined in Evan’s original proposal. I think they could potentially get through a law prohibiting US businesses from using models developed without alignment research. But that’s on the original proposal, not the pause you were asking about.
Ah ok here’s something that’s implicit, that I can now make explicit: IF you’re taking big actions (e.g. Anthropic doing frontier AGI research) AND your justifications for doing that rely on a quite unclear claim X, THEN you should have reasoned out X pretty well and defend it publicly OR ELSE you probably are deceiving yourself and/or others about why you’re doing what you’re doing. (There’s plenty of exceptions, but still.)
Sorry, I wasn’t fully explicit. When I said “Advocate for a global stop on AGI research”, I meant for us (broadly, including Evan, say) to advocate to world governments that they should institute a global stop, not that anyone’s supposed to persuade labs to stop. (I mean, separately I’d like to push on many points of intervention, including persuading AGI researchers to stop it.)
I’m really not. Not sure why you think that. I can try to lay more things out, though I think I’ve indicated some / most of this? Anyway:
I think it’s plausible that a global stop could be coordinated. I’m a layman, so I can’t, like, argue for this super well or coherently.
I think that it’s bad for labs to be doing AGI research.
Anthropic seems to justify pushing the frontier by saying a global pause is probably not doable.
I have literally never heard a serious coherent case laid out that this is not doable. It’s just something that it’s embarrassing if you don’t understand. But that’s garbage reasoning.
If I had to guess, I would guess that most of them (non-leadership Anthropic employees) broadly are just mindfucked somehow. However, I see plenty of bits of apparent good faith and good intentions, I know some of the people there and think they are smart and have or had good intentions, etc. Also, generally it seems rude to just assume there is no such case with checking. So, I have been trying to check by asking some Anthropic employees, and a bit of searching, and asking on LW. So far, no one has said “oh yeah here’s a link”, but people continue to argue or otherwise act as though it’s obvious. You tell me what I’m supposed to make of this! Maybe this is what comes off as trolling? Maybe we really should unban Said!
Gotcha. That all makes sense. I agree on almost every point. I definitely think we should be advocating for a global stop on AGI research, as well as slowdowns, as well as working on alignment research as fast as we possibly can, while trying to avoid advancing capabilities faster than alignment. These aren’t at all mutually exclusive. Everyone who sees the problem should be doing all of them IMO. I haven’t seen strong arguments against putting effort into all of these at once.
I think it IS obvious why it’s hard. Any international agreement is hard, and even people who think about p(doom) think it’s on the order of 10% and are mostly basically okay pushing ahead on those odds. This brings us to the “mindfucked somehow”. I think that’s probably basically right. I’ve laid out what I think are the relevant mechanisms, in a great deal of detail and with a great deal of empirical backing, in the post I referenced but didn’t explain: Motivated reasoning, confirmation bias, and AI risk theory. I wouldn’t use such a strong term because I don’t think it’s as strong as that implies—and calling people mindfucked is a great way to even further motivate them against you and everything associated with you.
Which brings us to the charge of socratic trolling. I certainly believe you when you say that’s not what you’re trying to do. But your intention has only a weak causal relationship to whether the effect is net-helpful or not. I had Said in mind when that term popped into my head. I followed that drama with great interest, and have very mixed feelings about it, but I think there’s a legitimate concern there. I think three years was way too long, and in the first go-round didn’t agree with Duncan Sabien’s calls for his banning (despite his amusingly biting the bullet and arguing it was good that Socrates was banished/killed for similar behavior). But I found Habryka’s arguments in Banning Said Achmiz (and broader thoughts on moderation) convincing in the face of Zach MD’s recent objections in Comment on “Banning Said Achmiz”. I’m not sure the time invested in all of those massive posts was worth it, but I feel I did exit with a little wisdom on the dynamics of what I’m going to keep calling socratic trolling, because it mixes those virtues and costs.
I believe you’re being completely sincere, and that you actually had very good reasons for asking that question, after this exchange. I think you’re correct that this assumption deserves to be questioned and debated.
Framing it as you did without context looks somewhat combative to my eye, and also seems like an isolated demand for rigor. So I stand by my initial assessment of “concerning and probably more harm than good”. I think it’s pretty easy to mitigate the downsides while keeping the upsides. Showing proof of effort by laying out the reasons you’re asking make it clear that you’re not trolling. And deliberately framing the question in a friendly way avoids the look of it being combative or adversarial, and thereby sparking more argument and “mindfucking” polarization (on both sides, obviously, since rationalists unfortunately still have feelings and motivations and halo/horns effects).
Anyway, this has been a useful exchange and I appreciate it. I think we’re all amateurs at what it takes to get a global treaty on pausing AI research; experts on international treaties are not at all experts on the argument that would drive it forward, and vice-versa.
Anthropic did explicitly say in their recent piece on RSI that they would start investigating how to get a pause or slowdown. It seems like other concerned parties like you and I should also chip in a little research here and there, even if it’s not our domain of direct expertise.
I usually have a pretty bad intuitive reaction to people telling me that my writing is “too combative”, but I don’t know why fully, and in a small fraction of the cases I end up agreeing with them later, so I won’t respond further. I do want to say what I just said, though; not sure why, but maybe, for example, I feel that it’s somehow unfair, though since I can’t explain how this is a very unreliable sense.
I think it’s a really important and central supposition, and apparently has not been rigorously defended anywhere! This is 100% definitely not what the concept of “isolated demand for rigor” is for! What other such things am I allegedly not demanding rigor for, that I ought to be? Or do you disagree / am I confused somehow?
Ok. (If you wanted to update me personally, you haven’t done that on this point.)
Until this point, I have been genuinely unsure if there’s simply a report / blog post / something that someone might just link me to, explaining the case!! But yes, I think you’re right, I now agree it would be better for a comment like my first to come with a few sentences explaining that it seems like there isn’t such a case, that a global stop seems plausibly feasible, that Anthropic seems super far from appropriately supporting that, and that they should.
Fair. I’m aware that people react negatively to being tone policed. Somewhat ironically, this can motivate people against the concept of motivated reasoning effects I think are pretty crucial to our understanding and therefore our survival. So I’ve avoided doing that for the most part, and regretted doing it. I pressed on because you didn’t immediately react badly. I’m still not sure if bringing this up is harmful or helpful.
But I have heard people from the developer side of the fence say that they find LW a hostile environment and have trouble engaging here even though they feel they should. And the tone of discussions here certainly look like tribal dynamics and polarization are happening. So I think this is an important topic, although I’m not at all sure I’m raising it the right way. That big MR post was my due diligence on making sure I know what I’m talking about; now I need to decide whether and how to push the issue more forcefully.
I expect I haven’t changed your mind because you haven’t read my careful research and arguments. I wouldn’t expect you to change your mind without evidence. Of course the evidence I present in that post isn’t airtight, but to me it looks awfully likely that being at least somewhat deliberately warm/nice is worthwhile if you want to win people over to your side. That seems particularly likely if there’s anything debatable at all. If the subject is really cut-and-dried I think you can push harder and succeed, but you still slow rate of progress if your approach is setting off enemy-warning signals in the audience’s brains.
Anyway, thanks for engaging that far. This is an important topic to me, and I haven’t really engaged on it here before. So I appreciate it.
On your other point: I am also unsure if there’s a report or blog post that more thoroughly lays out the case for pause being impossible. I doubt there’s a particularly good one.
The inverse of such a report is a careful arguement for how it is possible. I’ve laid out bits and pieces of a case for how slowdown is quite plausible, even possibly as a default on the current trajectory, but not in one coherent place or for a full pause. But the arguments can be extended for the possiblity of a pause.
I won’t dive into that further right now, but I think it is a worthy collaborative project for LWers.
This makes sense. I will note however that when I (one time) asked an Anthropic employee about inviting someone over to their offices to explain / argue more in depth some crucial point (I forget; I think alignment difficulty), they said something like “last one or two times we tried that, the guest was dismissive”. So like, it looks a whole lot more like the crux is self-insulation, even if there is also undue hostility on LW. But, that is N=1. (I have other Ns that look like self-insulation, though of course that’s almost inherently ambiguous and my total N is small.)
(I do think in past I’ve at least watched from the sidelines, or even slightly participated in, arguably-undue dogpiley polite arguing, if not hostility.)
Definitely agree. (Cf. https://www.lesswrong.com/posts/Sdrzo7z3STzdrnwKW/what-exactly-would-an-international-ai-treaty-say-is-a-bad and https://www.lesswrong.com/posts/X9Z9vdG7kEFTBkA6h/what-could-a-policy-banning-agi-look-like )
Interesting. Both of those posts have the form of “what would an agreement say” which I think is totally missing the hard part. So I think that points at the answer to your original question, and why others regard it as obvious and you do not.
The answer is “because there’s no political will”. And so the question isn’t what would an agreement say, it’s where would the political will come from.
My answer is that the political will will come from AI progress, particularly from visible job loss and from human-seeming AI systems, which will trip pattern-matching to strange humans, which we intuitively regard as quite dangerous. Xenophobia exists for a reason; strange humans have been among our biggest dangers since the start of evolution. I’ve written about this in A country of alien idiots in a datacenter: AI progress and public alarm and bits and pieces elsewhere.
On the positive side, this answer is that the political will will come, which shifts the question back to having an agreement or treaty ready to offer.
The problem with this answer is that the will might come too late. By the time systems are visibly taking jobs and acting agenticly and competently, we might already have a takeover-capable system in development, and it will be too late for anything as slow as international agreements. There still might be time for an executive order and informal power grabs in the face of adequate public (and politician) freakout. And that might be substantially useful, since only China is near the US, and they’d probably take a much more cautious approach to AGI and alignment (see China won’t win the AI race but would it be much worse if it did? and similar). I’ve written about this in Whether governments will control AGI is important and neglected and I now think the answer is just clearly yes, but maybe not in time.
That’s a slowdown not a pause, but it could perhaps be expanded to a pause if the discussion can move quickly between the US and China. I think this is possible. We’re really not enemies, just competitors. And our researchers are quite well-disposed toward each other.
Anyway, I think the major crux between us and the rest of the world is alignment risk. The average belief even among those that acknowledge the risk (which tbf is now pretty much anyone thinking about AGI) is maybe ten percent or lower. That’s enough to make some people want to pause, but not I think most of them, and not enough to make it a high priority.
So I think the clearest path toward pause (or slowdown) is clearer arguments about alignment risk.
Here I think overclaiming on the technical arguments has done grevious harm to the cause. Yudkowsky’s claim that misalignment is 99%+ likely has drawn much irritation and ire and attention. People, even sophisticated people, routinely argue “alignment is possible” instead of arguing about how possible. I think they’re quite correct that Yudkowsky’s technical argument is full of holes, but quite wrong in the implied leap from there to ~10% risk. Alignment may be quite achievable and still quite difficult to achieve on the current rushed path.
I think arguments for human incompetence on first tries and under pressure are a much better bet. To his credit, EY has shifted hard in this direction, and so have the handful of others making technical arguments for alignment difficulty. My arguments center on model uncertainty: nobody knows how hard alignment is; estimates from people with real time-on-task range from very low to very high; therefore the wisest assumption is that it could be extremely difficult and we are foolish to press ahead with so much unknown.
Here I think we could do vastly better. Optimists reason that current systems seem pretty aligned, so we’re probably on track to align more powerful systems. Pessimists argue that this isn’t useful evidence at all. Identifying cruxes and improving models of likely first AGI seems quite achievable, so that’s what I’m primarily working toward and asking others to engage in.
WRT the Anthropic office visits: This has the general form of “it’s their fault not ours” which is suspicious. In most disagreements, both parties blame the other. And even if it is totally their fault, I’d rather survive than assign blame. Usually the way forward in resolving interpersonal issues is “sorry about that, let’s try again” and then be nicer.
This is when you’re trying to reach mutual agreement with someone, not when you’re trying to negotiate a deal and have some leverage. Discussions about beliefs only resemble negotiations when the evidence is overwhelming. And on the dangers of alignment, it’s just unfortunately not.
You might be overinferring what I think these blog posts indicate? I’m just gesturing that I agree that the overall project of figuring out how the whole thing might be feasible is a worthy project.
I know that this is a thing people say, and I agree there isn’t already automatically political will pre-gathered. But if the implication is that it would be an infeasible task to create and gather the political will for a global stop, that implication is one I strongly question! And so far I hear lots of signs pointing in the opposite direction, and grateful to the people working on that. I just wish that Anthropic would support those efforts.
Not blaming, describing. Can’t survive without describing.
(Anyway, just FYI, your time might be somewhat wasted if you want to get me on board with a particular approach / stance, because I’m much more commenting from the sidelines rather than an active participant; I’m focusing on other things, while others are actually working on communicating with the public and political leaders and so on.)
That’s fine, I’ll consider it workshopping.
I’m also primarily occupied with other things. I’m spending some time on communication strategy and the logic of how opinion and policy could change, because it seems like it could be critically important, and not enough people seem to be thinking about it. As you note.
I hope you will too.
Oh—I left out one important consideration in my theory of how people get “mindfucked” in Motivated reasoning, etc: social dynamics. When I dug into the logic and the science deeply, I wound up thinking that social networks of who you respect and who they respect, and the resultant “double-counting”—actually many-counting—are probably even stronger effects than the strong individual effects I could put at least approximate empirically supported numbers to. (I actually think you can measuer network effects, I just had spent way too long on the project to dive into another area of research I wasn’t already familiar with). Anyway: mutual respect dynamics are probably strong, and they’re probably influenced strongly by motivated reasoning/horns&halo effects.
This is another reason to make sure to avoid polarization and an us-vs-them attitude. It makes everyone more mindfucked, in multiple and empirically strong ways.
I agree re/ networks (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html). I also agree with polarization being bad. However:
Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
I think this means something like, you strongly strongly criticize the behavior, but not demonize the person. I have for example said this: https://www.lesswrong.com/posts/CYTwRZtrhHuYf7QYu/a-case-for-courage-when-speaking-of-ai-danger?commentId=pLH6dxnTrTz56BQYj
I’m curious how else we-broadly can go about this better.
Something I’ve thought before is that it seems like most people are rolling their own conclusion about the political feasibility of pausing. They think about it for five minutes or less, and then they’re done; they decide whether building a US pause coalition sounds reasonable, or whether China could be cooperated with, mostly on priors. There’s no Rootclaim for the politics of an AI pause. No one org owns a pipeline for doing this research, not even for the narrow version of message testing. There’s just PauseAI, ControlAI, and StopAI doing their own scattered advocacy efforts live, with close to no support. Vastly influential decisions that caused hundreds of millions to flow to technical safety, and at most a few million to advocacy, were made mostly on vibes. No one even did A/B testing or focus groups for pause messaging until last year!
Mind that I don’t mean the technical implementation or effects of a pause, like with what MIRI does or what the 2023 AI Pause Debate did. I mean whether it seems politically achievable, whether it’s even in the Overton window, or if the window can be moved there. We saw with how PEPFAR was founded that sometimes a political miracle can just happen if there’s the will for it.
I think the question of “is a pause politically feasible” should have an adversarial collaboration done on it. That’s one of the best methods of truth-finding I know of, and it’s sad we don’t do more of it.
I suspect this may be the case, but I’d like to know how people think about their beliefs about this question, but the way Anthropic employees (n=3ish) act when I try to ask about it is confusing. I wish they would just say “I’m relying on the judgement of so-and-so” or something like that, as it would be much clearer. (Cf. https://www.lesswrong.com/posts/zLG2DnJw6oEZqAgaE/tools-for-deferring-gracefully )
By “infeasible”, do you mean strictly “technically infeasible / infeasible even if there’s the will to make it happen”, or something that also includes “tractable / politically palatable”?
Feasible overall, so including political feasibility.
I like this! Here’s a specific concrete version of it just to have some numbers on the table:
Companies are legally required to allocate their compute as follows:
1. 60% on serving customers.
2. 15% on internal safety research that’s fully transparent to the public.
3. 10% on external safety research that’s fully transparent to the public & has access to the latest internal models.
4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)
Suppose this were a US law, that applies to all the major AI companies but not to e.g. Chinese companies or Mistral for example.
15% is less than the roughly 50% they seem to be spending now, so overall the pace of AI progress would slow down to maybe something like half the speed it’s going now. But the amount of compute devoted to capabilities progress would still be higher at each US company than any Chinese company for example, so US companies would stay in the lead.
Meanwhile the amount of internal compute going to safety research would double or quadruple, and then the effect size of a ton of external researchers being able to look at the work, critique it, run their own replications and ablations, run their own experiments, etc. would be way way bigger on top of that already big positive effect.
How would it be enforced? Like you said, whistleblowers. If we were trying to ban AI capabilities R&D entirely, that would be super difficult, but if we are just keeping it to 15% of the compute, then cheaters can’t really prosper. It’s really hard to hide 10% of your compute being spent on something it’s not supposed to be spent on; maaayybe you can hide 1% but that wouldn’t be enough to make much difference & so wouldn’t be worth the risk.
This would also prevent a sort of race to the bottom where companies use more and more of their compute on R&D instead of serving customers; instead we’d have the process of AI deployment/diffusion continuing to happen which I think is broadly pretty good for the world.
I don’t feel strongly about the specific numbers above obviously but yeah this seems promising.