Separately, do you think “organized opposition” could have ever been avoided? It sounds like you’re making two claims:
When AI safety folks advocate for specific policies, this gives opponents something to rally around and makes them more likely to organize.
There are some examples of specific policies (e.g., restrictions on OS, SB1047) that have contributed to this.
Suppose no one said anything about OS, and also (separately) SB1047 never happened. Presumably, at some point, som groups start advocating for specific policies that go against the e/acc worldview. At that point, it seems like you get the organized resistance.
So I’m curious: What does the Ideal Richard World look like? Does it mean people are just much more selective about which policies to advocate for? Under what circumstances is it OK to advocate for something that will increase the political organization of opposing groups? Are there examples of policies that you think are so important that they’re worth the cost (of giving your opposition something to rally around?) To what extent is the deeper crux the fact that you’re less optimistic about the policy proposals actually helping?
Presumably, at some point, some groups start advocating for specific policies that go against the e/acc worldview. At that point, it seems like you get the organized resistance.
My two suggestions:
People stop aiming to produce proposals that hit almost all the possible worlds. By default you should design your proposal to be useless in, say, 20% of the worlds you’re worried about (because trying to get that last 20% will create really disproportionate pushback); or design your proposal so that it leaves 20% of the work undone (because trusting that other people will do that work ends up being less power-seeking, and more robust, than trying to centralize everything under your plan). I often hear people saying stuff like “we need to ensure that things go well” or “this plan needs to be sufficient to prevent risk”, and I think that mindset is basically guaranteed to push you too far towards the power-seeking end of the spectrum. (I’ve added an edit to the end of the post explaining this.)
As a specific example of this, if your median doom scenario goes through AGI developed/deployed by centralized powers (e.g. big labs, govts) I claim you should basically ignore open-source. Sure, there are some tail worlds where a random hacker collective beats the big players to build AGI; or where the big players stop in a responsible way, but the open-source community doesn’t; etc. But designing proposals around those is like trying to put out candles when your house is on fire. And I expect there to be widespread appetite for regulating AI labs from govts, wider society, and even labs themselves, within a few years’ time, unless those proposals become toxic in the meantime—and making those proposals a referendum on open-source is one of the best ways I can imagine to make them toxic.
(I’ve talked to some people whose median doom scenario looks more like Hendrycks’ “natural selection” paper. I think it makes sense by those people’s lights to continue strongly opposing open-source, but I also think those people are wrong.)
I think that the “we must ensure” stuff is mostly driven by a kind of internal alarm bell rather than careful cost-benefit reasoning; and in general I often expect this type of motivation to backfire in all sorts of ways.
Why do you assume that open source equates to small hacker groups??? The largest supplier of open weights is Meta AI and their recent Llama-405B rivals SOTA models.
I think your concrete suggestions such as these are very good. I still don’t think you have illustrated the power-seeking aspect you are claiming very well (it seems to be there for EA, but less so for AI safety in general).
In short, I think you are conveying certain important, substantive points, but are choosing a poor framing.
Separately, do you think “organized opposition” could have ever been avoided? It sounds like you’re making two claims:
When AI safety folks advocate for specific policies, this gives opponents something to rally around and makes them more likely to organize.
There are some examples of specific policies (e.g., restrictions on OS, SB1047) that have contributed to this.
Suppose no one said anything about OS, and also (separately) SB1047 never happened. Presumably, at some point, som groups start advocating for specific policies that go against the e/acc worldview. At that point, it seems like you get the organized resistance.
So I’m curious: What does the Ideal Richard World look like? Does it mean people are just much more selective about which policies to advocate for? Under what circumstances is it OK to advocate for something that will increase the political organization of opposing groups? Are there examples of policies that you think are so important that they’re worth the cost (of giving your opposition something to rally around?) To what extent is the deeper crux the fact that you’re less optimistic about the policy proposals actually helping?
My two suggestions:
People stop aiming to produce proposals that hit almost all the possible worlds. By default you should design your proposal to be useless in, say, 20% of the worlds you’re worried about (because trying to get that last 20% will create really disproportionate pushback); or design your proposal so that it leaves 20% of the work undone (because trusting that other people will do that work ends up being less power-seeking, and more robust, than trying to centralize everything under your plan). I often hear people saying stuff like “we need to ensure that things go well” or “this plan needs to be sufficient to prevent risk”, and I think that mindset is basically guaranteed to push you too far towards the power-seeking end of the spectrum. (I’ve added an edit to the end of the post explaining this.)
As a specific example of this, if your median doom scenario goes through AGI developed/deployed by centralized powers (e.g. big labs, govts) I claim you should basically ignore open-source. Sure, there are some tail worlds where a random hacker collective beats the big players to build AGI; or where the big players stop in a responsible way, but the open-source community doesn’t; etc. But designing proposals around those is like trying to put out candles when your house is on fire. And I expect there to be widespread appetite for regulating AI labs from govts, wider society, and even labs themselves, within a few years’ time, unless those proposals become toxic in the meantime—and making those proposals a referendum on open-source is one of the best ways I can imagine to make them toxic.
(I’ve talked to some people whose median doom scenario looks more like Hendrycks’ “natural selection” paper. I think it makes sense by those people’s lights to continue strongly opposing open-source, but I also think those people are wrong.)
I think that the “we must ensure” stuff is mostly driven by a kind of internal alarm bell rather than careful cost-benefit reasoning; and in general I often expect this type of motivation to backfire in all sorts of ways.
Why do you assume that open source equates to small hacker groups??? The largest supplier of open weights is Meta AI and their recent Llama-405B rivals SOTA models.
I think your concrete suggestions such as these are very good. I still don’t think you have illustrated the power-seeking aspect you are claiming very well (it seems to be there for EA, but less so for AI safety in general).
In short, I think you are conveying certain important, substantive points, but are choosing a poor framing.