But I think we’d also have a higher chance of success if the strategy for fixing peoples’ incentives was more systematic.
Yes agreed but it’s so far from where this community is and IMO impossible at this point. EA/LW has no inclusion/exclusion criteria. There is no actual community. It’s a messy network of ideas and relationships and money. Even if 80% of us bought into some constraint, if anthropic decides to do 5B of philanthropy next year who do you think is getting the money? The people who wine and dine them or those with already status in some way (which may or may not be correlated with things we care about).
Then those people will have jobs to give out, or a high position in government/business, and the new grads will drift towards those people, because they need jobs and want status, and in a few years, we will be the greasy guys with no job attending rationalist meetings in a tier 2 city. If you complain at that point you will look like a loser who is just salty you couldn’t get the jobs.
Like I really really care about AI safety, and I still needed lighthaven/PDKU to give me a grant and an official title to really do anything significant. The pull towards “having a title for the thing I’m doing that I can explain to new people I meet and my mom and I make enough money to survive or relative to expectations” is just such a strong attractor state.
The world is unfair and it is very very hard to create a social movement that doesn’t get eaten from the inside from these dynamics (don’t capture what you can’t defend?) and if you really want to trace the blame all the way back, I blame the founders of EA and Lesswrong for thinking you could make a boundaryless structureless social movement that could defend it’s original values. This is an incredibly hard thing to do, and they weren’t particularly close to hitting the mark. I think my issue with OP is that while I generally agree with a lot of the micro assertions he is making, the incentives of these communities always pointed in these directions. I think if MacAskill/Yud/ other early people in the movement had more direct experience in community organizing and politics, and less as philosopher kings, then many things about EA/LW would be very different, and we wouldn’t have these issues, although we probably would have missed out on many of the good things too.
Also there will still be improvements even if we never have model better than current frontier. Harnesses, system prompts, apis that work well with agents, company integrations with AI and people’s skill at using them will all continue improving. It’s really hard to know how much additional capabilities could come from things listed or new similar innovations on top of them. Probably not relevant for the RSI takeover concerns but for the more banal concerns like bioterrorism and hacking and severe economic disruption.