I think your model of co-option dynamics here is non-obvious to me, and I currently think it’s (probably) false for AI safety though I might just not be understanding it.
In particular in adversarial situations I expect group coherence/coordination to make cooption harder rather than easier.
Like in general when I think through the logic of collective action and relax some of the game theory assumptions, or think through history when there’s powerful vs less powerful actors in (often violent) conflicts, people being less coordinated usually makes them more susceptible to cooption rather than less. Think of the conquistadors playing up intra-Americas conflict in the Americas, the Achaemenid Empire sponsoring intra-Greek conflicts, broadly divide-and-conquer tactics in history, etc.
As I mentioned in a bunch of comments, I think “loose social movement that centrally coordinates via vibes and whoever is currently the highest-prestige thought-leader” is a format that is both low on coherence/coordination and high on ability to be co-opted.
Indeed I would feel a lot better about AI Safety/EA/Rationality on a lot of these dimensions if there was more formal membership, more things like courts, etc.
The worry here is that we have chosen a fundamental form of social organization that scores low on defensibility and high on resource aquisition, and moving away from that is now very difficult. Many alternatives (both in the “less coordinated” and the “more coordinated” directions) seem to me to be more defensible here.
Or to phrase it in the terms of the post:
I wish we either conquered less, or had more of a plan for how to defend what we conquered. Right now we are doing a lot of conquering, but without any plan for how to defend it, and that seems like it really has a pretty high chance of going badly.
Literal courts are expensive, but there’s a larger design space if we relax the constraints. Courts have to scale to state-size, interoperate with a huge variety of participants (lawyers, judges, police, other officers, and arbitrary citizens), be robust to certain kids of adversarial attacks, …
My cached thought is to have a norm of “if you’re occupying an important exclusive social niche, such as company leader, thought leader, etc., then you have an obligation to debate representatives from major disagreeing relevant views”. May require infrastructure for better debates to go well.
I think your model of co-option dynamics here is non-obvious to me, and I currently think it’s (probably) false for AI safety though I might just not be understanding it.
In particular in adversarial situations I expect group coherence/coordination to make cooption harder rather than easier.
Like in general when I think through the logic of collective action and relax some of the game theory assumptions, or think through history when there’s powerful vs less powerful actors in (often violent) conflicts, people being less coordinated usually makes them more susceptible to cooption rather than less. Think of the conquistadors playing up intra-Americas conflict in the Americas, the Achaemenid Empire sponsoring intra-Greek conflicts, broadly divide-and-conquer tactics in history, etc.
As I mentioned in a bunch of comments, I think “loose social movement that centrally coordinates via vibes and whoever is currently the highest-prestige thought-leader” is a format that is both low on coherence/coordination and high on ability to be co-opted.
Indeed I would feel a lot better about AI Safety/EA/Rationality on a lot of these dimensions if there was more formal membership, more things like courts, etc.
The worry here is that we have chosen a fundamental form of social organization that scores low on defensibility and high on resource aquisition, and moving away from that is now very difficult. Many alternatives (both in the “less coordinated” and the “more coordinated” directions) seem to me to be more defensible here.
Or to phrase it in the terms of the post:
I wish we either conquered less, or had more of a plan for how to defend what we conquered. Right now we are doing a lot of conquering, but without any plan for how to defend it, and that seems like it really has a pretty high chance of going badly.
Good point.
Literal courts are expensive, but there’s a larger design space if we relax the constraints. Courts have to scale to state-size, interoperate with a huge variety of participants (lawyers, judges, police, other officers, and arbitrary citizens), be robust to certain kids of adversarial attacks, …
My cached thought is to have a norm of “if you’re occupying an important exclusive social niche, such as company leader, thought leader, etc., then you have an obligation to debate representatives from major disagreeing relevant views”. May require infrastructure for better debates to go well.