The authors propose to get an international treaty to pause progress towards superintelligence, including both scaling & R&D. I’m for it, although I don’t hold out much hope for such efforts to have more than marginal impact. I expect that AI capabilities would rebrand as AI safety, and plow ahead:
The problem is: public advocacy is way too centered on LLMs, from my perspective. Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!
[...]
I think these people are generally sincere but mistaken, and I expect that, just as they have fooled themselves, they will also successfully fool their friends, their colleagues, and government regulators…
This seems way too pessimistic to me. (Or like sure it’s going to be hard and I’m not super optimistic, but given that you’re also relatively pessimistic the international AI R&D shutdown approach doesn’t seem too unpromising to me.)
Sure they are going to try to convince government regulators that their research is great for safety, but we’re going to try to convince the public and regulators otherwise.
I mean it’s sorta understandable to say that we currently seem to be in a relatively weak position and getting sufficient change seems hard, but movements can grow quickly. Yeah understandable that this doesn’t seem super convincing, but I think we have a handful of smart people who might be able to find ways to effectively shift the gameboard here. Idk.
More to the point though, conditional that we manage to internationally ban AI R&D, it doesn’t obviously seem that much more difficult or that much less likely that we manage to also ban AI safety efforts which can lead to AI capability increases, based on the understanding that those efforts are likely delusional and alignment is out of reach. (Tbc I would try to not ban your research, but given that your agenda is the only one I am aware of into which I put significantly more than 0 hope, it’s not clear to me that it’s worth overcomplicating the ban around that.)
Also in this common knowledge problem domain, self-fulfilling prophecies are sorta a thing, and I think it’s a bit harmful to the cause if you post on twitter and bluesky that you don’t have much hope in government action. Tbc, don’t say the opposite either, keep your integrity, but maybe leave the critizism on lesswrong? Idk.
This seems way too pessimistic to me. (Or like sure it’s going to be hard and I’m not super optimistic, but given that you’re also relatively pessimistic the international AI R&D shutdown approach doesn’t seem too unpromising to me.)
Sure they are going to try to convince government regulators that their research is great for safety, but we’re going to try to convince the public and regulators otherwise.
I mean it’s sorta understandable to say that we currently seem to be in a relatively weak position and getting sufficient change seems hard, but movements can grow quickly. Yeah understandable that this doesn’t seem super convincing, but I think we have a handful of smart people who might be able to find ways to effectively shift the gameboard here. Idk.
More to the point though, conditional that we manage to internationally ban AI R&D, it doesn’t obviously seem that much more difficult or that much less likely that we manage to also ban AI safety efforts which can lead to AI capability increases, based on the understanding that those efforts are likely delusional and alignment is out of reach. (Tbc I would try to not ban your research, but given that your agenda is the only one I am aware of into which I put significantly more than 0 hope, it’s not clear to me that it’s worth overcomplicating the ban around that.)
Also in this common knowledge problem domain, self-fulfilling prophecies are sorta a thing, and I think it’s a bit harmful to the cause if you post on twitter and bluesky that you don’t have much hope in government action. Tbc, don’t say the opposite either, keep your integrity, but maybe leave the critizism on lesswrong? Idk.