Oh—I left out one important consideration in my theory of how people get “mindfucked” in Motivated reasoning, etc: social dynamics. When I dug into the logic and the science deeply, I wound up thinking that social networks of who you respect and who they respect, and the resultant “double-counting”—actually many-counting—are probably even stronger effects than the strong individual effects I could put at least approximate empirically supported numbers to. (I actually think you can measuer network effects, I just had spent way too long on the project to dive into another area of research I wasn’t already familiar with). Anyway: mutual respect dynamics are probably strong, and they’re probably influenced strongly by motivated reasoning/horns&halo effects.
This is another reason to make sure to avoid polarization and an us-vs-them attitude. It makes everyone more mindfucked, in multiple and empirically strong ways.
Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
Oh—I left out one important consideration in my theory of how people get “mindfucked” in Motivated reasoning, etc: social dynamics. When I dug into the logic and the science deeply, I wound up thinking that social networks of who you respect and who they respect, and the resultant “double-counting”—actually many-counting—are probably even stronger effects than the strong individual effects I could put at least approximate empirically supported numbers to. (I actually think you can measuer network effects, I just had spent way too long on the project to dive into another area of research I wasn’t already familiar with). Anyway: mutual respect dynamics are probably strong, and they’re probably influenced strongly by motivated reasoning/horns&halo effects.
This is another reason to make sure to avoid polarization and an us-vs-them attitude. It makes everyone more mindfucked, in multiple and empirically strong ways.
I agree re/ networks (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html). I also agree with polarization being bad. However:
Often I think “fear of polarization” ends up making the person not able to ask tough questions at all; and sometimes they end up straight up going over to work on bad stuff.
People working on bad stuff are absolutely taking advantage of orientations like “fear of polarization”. (Not everyone, and it’s a mix of sympathetic / unsympathetic, intentional / unintentional; but still happening.) For example, I suspect this is a primary enabler of self-deception—being unclear about what’s wrong about someone’s beliefs or actions.
I think this means something like, you strongly strongly criticize the behavior, but not demonize the person. I have for example said this: https://www.lesswrong.com/posts/CYTwRZtrhHuYf7QYu/a-case-for-courage-when-speaking-of-ai-danger?commentId=pLH6dxnTrTz56BQYj
I’m curious how else we-broadly can go about this better.