Thanks, Tsvi! We appreciate you bringing in constructive points like this. Sure, the framing of this group feels kinda deliberately overcompressed, but the core point warrants serious consideration. We’ve touched on this a bit in our post, and we’re keen to break this down further in a future write-up, but I’ll say a few things in response to your comment (and the ensuing thread).
We shouldn’t just assume that being driven by good intentions means action is better than inaction by default (I’m mostly saying this in response to @clone of saturn’s reply to you in this thread). I do think we have a burden of proof in establishing the robustness of our proposed intervention.
We’ve laid out some of our models in this post partly to let you examine our case for this and hopefully resource and refine it. We want to have as much perspective and expertise on this as possible so we can build the best movement possible together.
“These plans about social support for global AGI bans involve creating and coalescing a big blob of social/political will
This nicely highlights what we’re not building and why. The “blob” is an amorphous mass of people who occasionally get “activated” into taking a small action, or a movement people join because they share dissent about something without having a clear ask to realise (e.g. Occupy Wall Street). The blob
1) doesn’t demonstrate depth of commitment to decision makers,
2) doesn’t build a sustainable movement because people seem to lose momentum when they feel like they’re just getting borrowed to do favours to something they’re not meaningfully contributing to build,
3) doesn’t create long-term buy-in because it lacks clarity and vision, and
4) has no structure, i.e. lacks the coherence, responsiveness and effectiveness that comes with organisation (see our passage on how a human body grows, Part II.4).
We’re not claiming building the non-naive version of this is easy, but we do think that it’s possible to design a movement such that a blob is not what you get, and to thereby guard against the blob’s failure modes.
What are they gathering political will towards? Many of them are hoping to make a blob for “international treaty to prevent AGI research”, and they are saying that’s what they’re aiming for, and in the mental sense they are trying to do so. But in the likely-actual-results sense, they aren’t working towards that. What they are actually working towards is [whatever is desired by the highly competent political actors who will coopt and wield the blob].
This is a valid and important concern if the movement is a political actor that is incompetent /vulnerable to cooptation, which is why we want to secure the resources to build the competent version, and why we want to work with the rest of our community to do this well.
The frontier AI industry and their political partners contain some very competent people (to the extent that you can call driving humanity towards a cliff competence, so I’ll settle for ‘narrowly competent’). Luckily, there are other competent people out there. Some of them are helping build the foundations of this very movement. Many of them are doing other things—PSA: if that’s you, pls check if what you’re doing is higher-EV than helping us build the political counterforce to end the race; if you can help us get the resources to get 1, 10 or 100 more competent people to build this with us, pls do so. Cooptation is a real threat if we don’t proactively prevent it, but we think we can prevent it with good design. By good design, I mean things like:
- a clear ask that’s tamper-evident, underpinned by strong policy drafting on the part of our friends in the Moyer Reformer role (see Part II.1). In that sense, this is an ecosystemic failure mode that we need to watch for together, and a predictable one that good policy can prepare for. Let’s examine all the ways serious candidate proposals are vulnerable to subversion and mitigate for this;
- a balance of top-down and bottom-up power built on many solid links, which removes the vulnerability of relying only on weak/overstretched links in the chain/sparse nodes (bottom-up only) or on a single point of failure (top-down only);
- a culture that encodes the responsibility each member takes by embracing civic leadership (which I expect switches on people’s sense of watchfulness) and good epistemic hygiene (which encourages distributed critical thinking), AND teaches an awareness that industry’s Goliath will likely try very creative things to not have to put its toys in a box, including trying to persuade movement supporters that it is doing the things we need it to. I think this is a thing you can select for in hiring and pass on intentionally, almost constitutionally, to new hires and generations of organisers if you ensure your infrastructure is fit for purpose—training materials, contact with the movement’s culture via regular calls with chapter leads etc. You can nurture a movement people want to be a part of because it resonates with their values, empowers them, and gives them a community to be part of. I expect this creates a layer of robustness to attempts to waylay members, because I don’t think a PR machine will easily seduce people away from a community that treats them well and consistently shows a commitment to existential safety. Its tactics will feel hollow.
It seems attempts to astroturf counter-mobilisation are already getting exposed here and there, which should enable our members’ immune systems around this attack vector. As for the risk of leadership cooptation, we think our current structure (see Part II.2) offers some safeguards, and the people with decision-making power in our movement have demonstrated commitment to AI safety and the kind of integrity our values call for. There may be more we can do.
- a nonpartisan approach, which if well executed, can prevent wholesale capture of major segments of our movement’s members and wider public opinion (and here I think @clone of saturn is right to highlight that not having a movement in place is probably pretty negative EV).
I’ll limit myself to those examples rather than try and draw an exhaustive list here, because this context is too cramped to unpack this rather intricate object. The point is, there are things we can do to strongly mitigate the otherwise very real risk of cooptation, we should be doing them, we’re designing accordingly, and we want your help making our movement as robust as possible.
One thing I’ll gesture at in passing is that the engaged volunteers I know are deeply bought into the idea that humanity stands on the edge of a precipice; they volunteer on the side of their busy lives because they relentlessly remember their kids’ lives are at stake, and they are wary of what industry will try to get to prevent the world’s citizens from stopping it from pursuing dangerous R&D (industry has not shown itself to prioritise safety, and thereby has not given people incentives to trust them; I expect retro-manufacturing that trust would be pretty hard, though obviously I’m not suggesting we just rely on that and drop our guard).
The actors who are currently trying to create that political will are not highly competent political actors. In general, when there is a big blob of political will not already controlled/wielded by highly competent political actors, that blob is likely to soon come under the control of highly competent political actors.
I believe we have done promising work so far towards demonstrating political competence (see Part III for our track record), and I also think it’s perfectly legitimate for people not to know this and to expect we will need a very high level of competence as we step into our role more fully. This is why we want your help—we want to grow our team with world-class people so we can do the excellent job we need to do as we scale.
Hey @davekasten and @Ondřej_Kubů, I was hoping to get to more comments this week but I realise it makes sense to let Maxime reply when he’s back from well deserved leave, he knows the whole history a bunch better than me and since this is a delicate subject for some people, I’ll leave it with him, as our CEO, to reply. I promise we’re not forgetting about your comments.