I often hear people say that moderates end up too friendly to AI companies due to working with people from AI companies. I agree, but I think that working as a moderate has a huge advantage for your epistemics.
I think that this friendliness has its own very large epistemic effects. The better you know people, the more time you spend with them, and the friendlier you are with them, the more cognitive dissonance you have to overcome in order to see them as doing bad things, and especially to see them as bad people (in the sense of their actions being net harmful for the world). This seems like the most fundamental force behind regulatory capture (although of course there are other factors there like the prospect of later getting industry jobs).
You may be meaning to implicitly recognize this dynamic in the quote above; it’s not clear to me either way. But I think it’s worth explicitly calling out as having a strong countervailing epistemic impact. I’m sure it varies significantly across people (maybe it’s roughly proportional to agreeability in the OCEAN sense?), and it may not have a large impact on you personally, but for people to weigh the epistemic value of working as a moderate, it’s important that they consider this effect.
A related phenomenon: Right-leaning Supreme Court justices move left as they get older, possibly because they’re in a left-leaning part of the country (DC) and that’s where all their friends are.
A typical justice nominated by a Republican president starts out at age 50 as an Antonin Scalia and retires at age 80 as an Anthony Kennedy. A justice nominated by a Democrat, however, is a lifelong Stephen Breyer.
[…]
The Cocktail Scene. Maybe the justices — human as they are, after all — want to fit in at parties. “Justices may be subject to influences by the Beltway cocktail scene and want to be perceived as reasonable and moderate,” Josh Blackman, a Supreme Court scholar at the South Texas College of Law, told me in an email. That assumes the cocktail set is liberal, what with its law professors and journalists. But that stereotype does exist in D.C. President Richard Nixon, for example, explicitly wondered if his Supreme Court nominee Harry Blackmun could “resist the Washington cocktail party circuit.”
This is a significant effect in general, but I’m not sure how much epistemic cost it creates in this situation. Moderates working with AI companies mostly interact with safety researchers, who are not generally doing bad things. There may be a weaker second-order effect where the safety researchers at labs have some epistemic distortion from cooperating with capabilities efforts, and this can influence external people who are collaborating with them.
I think that this friendliness has its own very large epistemic effects. The better you know people, the more time you spend with them, and the friendlier you are with them, the more cognitive dissonance you have to overcome in order to see them as doing bad things, and especially to see them as bad people (in the sense of their actions being net harmful for the world). This seems like the most fundamental force behind regulatory capture (although of course there are other factors there like the prospect of later getting industry jobs).
You may be meaning to implicitly recognize this dynamic in the quote above; it’s not clear to me either way. But I think it’s worth explicitly calling out as having a strong countervailing epistemic impact. I’m sure it varies significantly across people (maybe it’s roughly proportional to agreeability in the OCEAN sense?), and it may not have a large impact on you personally, but for people to weigh the epistemic value of working as a moderate, it’s important that they consider this effect.
A related phenomenon: Right-leaning Supreme Court justices move left as they get older, possibly because they’re in a left-leaning part of the country (DC) and that’s where all their friends are.
[…]
This is a significant effect in general, but I’m not sure how much epistemic cost it creates in this situation. Moderates working with AI companies mostly interact with safety researchers, who are not generally doing bad things. There may be a weaker second-order effect where the safety researchers at labs have some epistemic distortion from cooperating with capabilities efforts, and this can influence external people who are collaborating with them.
Fair point, that does seem like a moderating (heh) factor.