people and governments taking AGI very seriously, something that is not the case right now and I believe is unlikely to become the case under a pause.
By a “pause”, the main thing I mean is “an international treaty to stop AGI globally”. I assume you’d think that’s unlikely because it’s unlikely that people + gvts would take it seriously enough. I don’t want to make a strong claim about it being likely feasible, and presumably even if it’s doable it would be a huge amount of hard work. But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
we are very close to RSI/AGI/ASI in which case I think a global pause does not slow progress enough (due to enforcement issues)
I mean, I agree that it’s kinda hard, but if AGI is very close, it probably involves lots of compute, right? Big piles of compute seem plausibly regulable. That doesn’t seem like an infeasible enforcement issue.
we are not very close to RSI/AGI/ASI in which case we should hold off on a hard pause because the case for it will be stronger in the future. …. although I do worry that a stop would lower the salience of the issue.
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
I don’t think you can talk about feasibility in a vacuum. Is it feasible that Congress pass a constitutional amendment removing the electoral college that is ratified by the states? From a technical sense, of course. From my perspective as someone who wants that to happen, not really.
Let me try to lay out my overall position.
I think right now you can break players into three categories
If all players are in group 1 (and knew the others were in group 1) a global pause is trivial. If all players are in groups 1 and 2 a global pause is a technical question: Can you create a treaty/system strong enough that secret defection is not possible? Players in group 3 have no interest in a pause and would have to be brought to the table with other incentives.
It’s very hard to know who is in which camp. I’d guess lab employees are largely split between 1, 2, and 3. I think the companies themselves are probably in groups 2 or 3.I think the U.S. government is in group 3 (caveat that there are lots of competing voices here and I have a very hard time modelling Trump). I have no idea where China or other world powers are.
The groups are more of a spectrum than binary. Someone in group 2 who thinks the status quo is very negative will be willing to spend more resources on a pause than someone who more loosely prefers a pause.
So I think the question of feasibility is how feasible is it for people in groups 1 and 2 to get the U.S. government into group 2 and strongly enough in group 2 that it is willing to expend lots of resources on a treaty (and I think since there are other players in group 3 you would need lots of resources). I don’t have a rock solid case that this is infeasible (I don’t think you ever really could), but I think it likely is at this time. EDIT: Adding in a little more here because I saw your conversations with others and think this may be our central disagreement. I think many people overrate how politically salient AI is. Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep. When you look at Gallup polling of the most important issues facing our country, AI doesn’t show up. I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished. I think if there are anti-AI legislation that is electorally valuable it will probably be surface level stuff that doesn’t move the needle on AI risk (e.g. data center water regulations). Now voters are not an immovable force, but I think it will be hard to move them quickly and may take some other shock to the system (impactful warning shot, widespread job loss, etc.)
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
As for why we might want to wait for a pause if AGI is further away, If we could have a strong pause now, I’d definitely agree with you. However, I think if you were to push the world into a weakly enforced pause it may be harder to get a strong pause down the line. In the status quo I hope we can slow down capabilities work, build up chip control systems and other monitoring options and as AI gets more powerful (hopefully before we all die) people will move towards a real pause. My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.
FWIW if I were a world dictator I would implement a global pause followed by heavily regulated and monitored technical alignment research. I also weakly hold the belief that people who think we are in an emergency situation should state that clearly and strongly, but have found some arguments for being more strategic compelling (e.g. Holden Karnofsky on the 80,000 hours podcast).
By a “pause”, the main thing I mean is “an international treaty to stop AGI globally”. I assume you’d think that’s unlikely because it’s unlikely that people + gvts would take it seriously enough. I don’t want to make a strong claim about it being likely feasible, and presumably even if it’s doable it would be a huge amount of hard work. But are you strongly claiming that it’s infeasible? If so, that’s the position I’d like to understand—why do you think that, if you do? Is this a case that’s been worked out and explained somewhere? Has it been debated seriously?
I mean, I agree that it’s kinda hard, but if AGI is very close, it probably involves lots of compute, right? Big piles of compute seem plausibly regulable. That doesn’t seem like an infeasible enforcement issue.
Neither of these arguments make sense to me, and seem quite opposite the truth. Like, we should stop ASAP so that we’re not in a terrible time crunch, right?
I don’t think you can talk about feasibility in a vacuum. Is it feasible that Congress pass a constitutional amendment removing the electoral college that is ratified by the states? From a technical sense, of course. From my perspective as someone who wants that to happen, not really.
Let me try to lay out my overall position.
I think right now you can break players into three categories
Those who model the situation as a stag hunt.
Those that model the situation as a prisoner’s dilemma.
Those that model the situation as a deadlock.
If all players are in group 1 (and knew the others were in group 1) a global pause is trivial. If all players are in groups 1 and 2 a global pause is a technical question: Can you create a treaty/system strong enough that secret defection is not possible? Players in group 3 have no interest in a pause and would have to be brought to the table with other incentives.
It’s very hard to know who is in which camp. I’d guess lab employees are largely split between 1, 2, and 3. I think the companies themselves are probably in groups 2 or 3.I think the U.S. government is in group 3 (caveat that there are lots of competing voices here and I have a very hard time modelling Trump). I have no idea where China or other world powers are.
The groups are more of a spectrum than binary. Someone in group 2 who thinks the status quo is very negative will be willing to spend more resources on a pause than someone who more loosely prefers a pause.
So I think the question of feasibility is how feasible is it for people in groups 1 and 2 to get the U.S. government into group 2 and strongly enough in group 2 that it is willing to expend lots of resources on a treaty (and I think since there are other players in group 3 you would need lots of resources). I don’t have a rock solid case that this is infeasible (I don’t think you ever really could), but I think it likely is at this time. EDIT: Adding in a little more here because I saw your conversations with others and think this may be our central disagreement. I think many people overrate how politically salient AI is. Anti-AI sentiment is all over the place, but I think its a mile wide and an inch deep. When you look at Gallup polling of the most important issues facing our country, AI doesn’t show up. I think any politicians that did serious damage to the U.S. economy and potentially started wars to pause AI would be electorally punished. I think if there are anti-AI legislation that is electorally valuable it will probably be surface level stuff that doesn’t move the needle on AI risk (e.g. data center water regulations). Now voters are not an immovable force, but I think it will be hard to move them quickly and may take some other shock to the system (impactful warning shot, widespread job loss, etc.)
As for why we might want to wait for a pause if AGI is further away, If we could have a strong pause now, I’d definitely agree with you. However, I think if you were to push the world into a weakly enforced pause it may be harder to get a strong pause down the line. In the status quo I hope we can slow down capabilities work, build up chip control systems and other monitoring options and as AI gets more powerful (hopefully before we all die) people will move towards a real pause. My concern is that a weak pause drives AI development underground, differentially hurts safety, and doesn’t allow people to update in the direction of a real pause. Like a think a world where AI development is nominally illegal, but the Chinese and U.S. Governments both had well funded secret programs is much worse than Evan’s proposal and likely worse than the status quo.
FWIW if I were a world dictator I would implement a global pause followed by heavily regulated and monitored technical alignment research. I also weakly hold the belief that people who think we are in an emergency situation should state that clearly and strongly, but have found some arguments for being more strategic compelling (e.g. Holden Karnofsky on the 80,000 hours podcast).