Yet there is still a cultural chasm between SF/SV and Berkeley AI safety people so you either missed the point of the comment or are dismissing the importance of this cultural divide, which is insane to me given how much the divide between the communities bears weight on current events.
Perhaps a useful proxy to measure the cultural divide is the proportion of openly poly people within SF/SV versus Berkeley AI safety circles?
I’ve been thinking about this a lot lately and for me the question has been partially reduced to: should or should not Constellation move to SF?
After reflection, my stance shifted to: probably not. Geographical isolation allows for a heterogenous (w/r/t SV) focus on safety, and enough of this culture trickles down to the frontier labs. The frontier labs in turn influence actions at the neo labs, which in turn influences the frontier labs themselves.
There is this broader question of how the community could shift the underlying incentives to punish reckless pursuit of capabilities. One thing that rationalists could have done is put more effort into shifting public opinion in their favor, but there are of course countless issues with politics and the bureaucracy involved. I would be interested in discussions between influential people in Silicon Valley and rationalists that are aimed towards thinking of institutions (whether non-profit, for-profit, or government) that could make the underlying incentives more favorable towards positive outcomes.
But, with respect to Berkeley alone, I think their role as an isolated rationalist chamber is probably a net positive. The counterfactual is having Buck, Geoffrey, Paul, and Beth in SF where they could perhaps better disseminate their pov to the frontier labs, but idk if this would actually work, and they may already have enough influence by, e.g, having relatively frictionless access to leading researchers.
I think that more effort should be put into thinking about for profit institutions with more favorable underlying dynamics whose culture can survive the stress of massive capital injections and scale. Creating a gov institution seems difficult (see CAISI—not sure what the post-mortem is there).
One of the issues about Berkeley proper is that I think a lot of the people in SF think that their focus on slow down of capabilities and their dismissal of capabilities research to be silly/unrealistic, and I think that alignment researchers should be more cognizant of this, though perhaps failure modes and catastrophes that results for irresponsible practices (like OAI HF incident) will just resolve this. I’ve heard that some OAI capabilities researchers are already shifting to more aligned-shaped research following the incident.
Still, I do think that rationalists/professionals would benefit from brainstorming sessions on how to shift incentives to benefit more thoughtful buildout of capabilities, though what outcomes are acceptable are also under contention.
Perhaps we can air-gap a model trained to reward-hack to find then patch exploits in training environments?