I appreciated the section that contrasted the “reasonable pragmatist strategy” with what people in the “pragmatist camp” sometimes seem to be doing.
EAs in AI governance would often tell me things along the lines of “trust me, the pragmatists know what they’re doing. They support strong regulations, they’re just not allowed to say it. At some point, when the time is right, they’ll come out and actually support meaningful regulations. They appreciate the folks who are pushing the Overton Window etc etc.”
I think this was likely wrong, or at least oversold. Maybe I was just talking to the wrong people, idk.
To me, it seems like we’re in an extremely opportune and important time for people in the “pragmatist camp” to come out and say they support strong regulations. Yoshua Bengio and Geoffrey Hinton and other respectable people with respectable roles are out here saying that this stuff could cause extinction.
I’m even fine with pragmatists adding caveats like “I would support X if we got sufficient evidence that this was feasible” or “I would support Y if we had a concrete version of that proposal, and I think it would be important to put thought into addressing Z limitation.”
They can also make it clear that they also support some of the more agreeable policies (e.g., infosec requirements). They can make it clear that some of the ambitious solutions would require substantial international coordination and there are certain scenarios in which they would be inappropriate.
Instead, I often see folks dismiss ideas that are remotely outside the Mainline Discourse. Concretely, I think it’s inappropriate for people to confidently declare that things like global compute caps and IAEA-like governance structures are infeasible. There is a substantial amount of uncertainty around what’s possible– world governments are just beginning to wake up to a technology that experts believe have a >20% of ending humanity.
We are not dealing with “normal policy”– we simply don’t have a large enough sample size of “the world trying to deal with existentially dangerous technologies” to be able to predict what will happen.
It’s common for people to say “it’s really hard to predict capabilities progress”. I wish it was more common for people to say “it’s really hard to predict how quickly the Overton Window will shift and how world leaders will react to this extremely scary technology.” As a result, maybe we shouldn’t dismiss the ambitious strategies that would require greater-than-climate-change levels of international coordination.
It isn’t over yet. I think there’s hope that some of the pragmatists will come out and realize that they are now “allowed” to say more than “maybe we should at least evaluate these systems with a >20% of ending humanity.” I think Joe Collman’s comment makes this point clearly and constructively:
The outcome I’m interested in is something like: every person with significant influence on policy knows that this is believed to be a good/ideal solution, and that the only reasons against it are based on whether it’s achievable in the right form.
If ARC Evals aren’t saying this, RSPs don’t include it, and many policy proposals don’t include it..., then I don’t expect this to become common knowledge.
We’re much less likely to get a stop if most people with influence don’t even realize it’s the thing that we’d ideally get.
I appreciated the section that contrasted the “reasonable pragmatist strategy” with what people in the “pragmatist camp” sometimes seem to be doing.
EAs in AI governance would often tell me things along the lines of “trust me, the pragmatists know what they’re doing. They support strong regulations, they’re just not allowed to say it. At some point, when the time is right, they’ll come out and actually support meaningful regulations. They appreciate the folks who are pushing the Overton Window etc etc.”
I think this was likely wrong, or at least oversold. Maybe I was just talking to the wrong people, idk.
To me, it seems like we’re in an extremely opportune and important time for people in the “pragmatist camp” to come out and say they support strong regulations. Yoshua Bengio and Geoffrey Hinton and other respectable people with respectable roles are out here saying that this stuff could cause extinction.
I’m even fine with pragmatists adding caveats like “I would support X if we got sufficient evidence that this was feasible” or “I would support Y if we had a concrete version of that proposal, and I think it would be important to put thought into addressing Z limitation.”
They can also make it clear that they also support some of the more agreeable policies (e.g., infosec requirements). They can make it clear that some of the ambitious solutions would require substantial international coordination and there are certain scenarios in which they would be inappropriate.
Instead, I often see folks dismiss ideas that are remotely outside the Mainline Discourse. Concretely, I think it’s inappropriate for people to confidently declare that things like global compute caps and IAEA-like governance structures are infeasible. There is a substantial amount of uncertainty around what’s possible– world governments are just beginning to wake up to a technology that experts believe have a >20% of ending humanity.
We are not dealing with “normal policy”– we simply don’t have a large enough sample size of “the world trying to deal with existentially dangerous technologies” to be able to predict what will happen.
It’s common for people to say “it’s really hard to predict capabilities progress”. I wish it was more common for people to say “it’s really hard to predict how quickly the Overton Window will shift and how world leaders will react to this extremely scary technology.” As a result, maybe we shouldn’t dismiss the ambitious strategies that would require greater-than-climate-change levels of international coordination.
It isn’t over yet. I think there’s hope that some of the pragmatists will come out and realize that they are now “allowed” to say more than “maybe we should at least evaluate these systems with a >20% of ending humanity.” I think Joe Collman’s comment makes this point clearly and constructively: