Someone else added these quotes from a 1968 article about how the Vietnam war could go so wrong:
Despite the banishment of the experts, internal doubters and dissenters did indeed appear and persist. Yet as I watched the process, such men were effectively neutralized by a subtle dynamic: the domestication of dissenters. Such “domestication” arose out of a twofold clubbish need: on the one hand, the dissenter’s desire to stay aboard; and on the other hand, the nondissenter’s conscience. Simply stated, dissent, when recognized, was made to feel at home. On the lowest possible scale of importance, I must confess my own considerable sense of dignity and acceptance (both vital) when my senior White House employer would refer to me as his “favorite dove.” Far more significant was the case of the former Undersecretary of State, George Ball. Once Mr. Ball began to express doubts, he was warmly institutionalized: he was encouraged to become the inhouse devil’s advocate on Vietnam. The upshot was inevitable: the process of escalation allowed for periodic requests to Mr. Ball to speak his piece; Ball felt good, I assume (he had fought for righteousness); the others felt good (they had given a full hearing to the dovish option); and there was minimal unpleasantness. The club remained intact; and it is of course possible that matters would have gotten worse faster if Mr. Ball had kept silent, or left before his final departure in the fall of 1966. There was also, of course, the case of the last institutionalized doubter, Bill Moyers. The President is said to have greeted his arrival at meetings with an affectionate, “Well, here comes Mr. Stop-the-Bombing....” Here again the dynamics of domesticated dissent sustained the relationship for a while.
A related point—and crucial, I suppose, to government at all times—was the “effectiveness” trap, the trap that keeps men from speaking out, as clearly or often as they might, within the government. And it is the trap that keeps men from resigning in protest and airing their dissent outside the government. The most important asset that a man brings to bureaucratic life is his “effectiveness,” a mysterious combination of training, style, and connections. The most ominous complaint that can be whispered of a bureaucrat is: “I’m afraid Charlie’s beginning to lose his effectiveness.” To preserve your effectiveness, you must decide where and when to fight the mainstream of policy; the opportunities range from pillow talk with your wife, to private drinks with your friends, to meetings with the Secretary of State or the President. The inclination to remain silent or to acquiesce in the presence of the great men—to live to fight another day, to give on this issue so that you can be “effective” on later issues—is overwhelming. Nor is it the tendency of youth alone; some of our most senior officials, men of wealth and fame, whose place in history is secure, have remained silent lest their connection with power be terminated. As for the disinclination to resign in protest: while not necessarily a Washington or even American specialty, it seems more true of a government in which ministers have no parliamentary backbench to which to retreat. In the absence of such a refuge, it is easy to rationalize the decision to stay aboard. By doing so, one may be able to prevent a few bad things from happening and perhaps even make a few good things happen. To exit is to lose even those marginal chances for “effectiveness.”
I’d be interested in a few more details/gears. (Also, are you primarily replying about the immediate parent, i.e. domestication of dissent, or also about the previous one)
Two different angles of curiosity I have are:
what sort of things you might you look out for, in particular, to notice if this was happening to you at OpenAI or similar?
something like… what’s your estimate of the effect size here? Do you have personal experience feeling captured by this dynamic? If so, what was it like? Or did you observe other people seeming to be captured, and what was your impression (perhaps in vague terms) of the diff that the dynamic was producing?
I was talking about the immediate parent, not the previous one. Though as secrecy gets ramped up, the effect described in the previous one might set in as well.
I have personal experience feeling captured by this dynamic, yes, and from conversations with other people i get the impression that it was even stronger for many others.
Hard to say how large of an effect it has. It definitely creates a significant chilling effect on criticism/dissent. (I think people who were employees alongside me while I was there will attest that I was pretty outspoken… yet I often found myself refraining from saying things that seemed true and important, due to not wanting to rock the boat / lose ‘credibility’ etc.
The point about salving the consciences of the majority is interesting and seems true to me as well. I feel like there’s definitely a dynamic of ‘the dissenters make polite reserved versions of their criticisms, and feel good about themselves for fighting the good fight, and the orthodox listen patiently and then find some justification to proceed as planned, feeling good about themselves for hearing out the dissent.’
I don’t know of an easy solution to this problem. Perhaps something to do with regular anonymous surveys? idk.
I wish this quote were a little more explicit about what’s going wrong. On a literal reading it’s saying that some people who disagreed attended meetings and were made to feel comfortable. I think it’s super plausible that this leads to some kind of pernicious effect, but I wish it spelt out more what.
I guess the best thing I can infer is that the author thinks public resignations and dissent would have been somewhat effective and the domesticated dissenters were basically ineffective?
Or is the context of the piece just that he’s explaining the absence of prominent public dissent?
My understanding is that institutions can create an internal “game of telephone” while believing that they are listening to the opposing opinions.
Basically, information gets diluted at each step:
you tell someone that “X is a problem”, but they understand it merely as “we need to provide some verbal justification why we are going to do X anyway”, i.e. they don’t treat it as an actual problem, but as a mere bureaucratic obstacle to be navigated around;
your friend tells you that “X is a huge problem”, but your takeaway is merely that “X is a problem”, because on one hand you trust your friend, on the other hand you think your friend is too obsessed with X and exaggerates the impact;
your friend had an inside information (which is the reason you have actually decided to listen to him in the first place) that X will probably kill everyone, but he realizes that if he tells it this way, you will probably decide that he is insane, so he instead chooses to put it diplomatically as “X is a huge problem”.
Taken together… you hired someone as an expert to provide an important information (and you congratulate yourself for doing it), but ultimately you ignored everything he said. And everyone felt happy in the process… your friend felt happy that the people making the important decisions were listening to him… your organization felt happy for having an ISO-certified process to listen to outside critics… except that ultimately nothing happened.
I am not saying that in a parallel reality where your friend instead wrote a Facebook post “we are all going to die” (and was ignored by everyone who matters) had a better outcome. But it had less self-deception by everyone involved.
Someone else added these quotes from a 1968 article about how the Vietnam war could go so wrong:
Wow, yeah. This is totally going on at OpenAI, and I expect at other AGI corporations also.
I’d be interested in a few more details/gears. (Also, are you primarily replying about the immediate parent, i.e. domestication of dissent, or also about the previous one)
Two different angles of curiosity I have are:
what sort of things you might you look out for, in particular, to notice if this was happening to you at OpenAI or similar?
something like… what’s your estimate of the effect size here? Do you have personal experience feeling captured by this dynamic? If so, what was it like? Or did you observe other people seeming to be captured, and what was your impression (perhaps in vague terms) of the diff that the dynamic was producing?
I was talking about the immediate parent, not the previous one. Though as secrecy gets ramped up, the effect described in the previous one might set in as well.
I have personal experience feeling captured by this dynamic, yes, and from conversations with other people i get the impression that it was even stronger for many others.
Hard to say how large of an effect it has. It definitely creates a significant chilling effect on criticism/dissent. (I think people who were employees alongside me while I was there will attest that I was pretty outspoken… yet I often found myself refraining from saying things that seemed true and important, due to not wanting to rock the boat / lose ‘credibility’ etc.
The point about salving the consciences of the majority is interesting and seems true to me as well. I feel like there’s definitely a dynamic of ‘the dissenters make polite reserved versions of their criticisms, and feel good about themselves for fighting the good fight, and the orthodox listen patiently and then find some justification to proceed as planned, feeling good about themselves for hearing out the dissent.’
I don’t know of an easy solution to this problem. Perhaps something to do with regular anonymous surveys? idk.
I wish this quote were a little more explicit about what’s going wrong. On a literal reading it’s saying that some people who disagreed attended meetings and were made to feel comfortable. I think it’s super plausible that this leads to some kind of pernicious effect, but I wish it spelt out more what.
I guess the best thing I can infer is that the author thinks public resignations and dissent would have been somewhat effective and the domesticated dissenters were basically ineffective?
Or is the context of the piece just that he’s explaining the absence of prominent public dissent?
My understanding is that institutions can create an internal “game of telephone” while believing that they are listening to the opposing opinions.
Basically, information gets diluted at each step:
you tell someone that “X is a problem”, but they understand it merely as “we need to provide some verbal justification why we are going to do X anyway”, i.e. they don’t treat it as an actual problem, but as a mere bureaucratic obstacle to be navigated around;
your friend tells you that “X is a huge problem”, but your takeaway is merely that “X is a problem”, because on one hand you trust your friend, on the other hand you think your friend is too obsessed with X and exaggerates the impact;
your friend had an inside information (which is the reason you have actually decided to listen to him in the first place) that X will probably kill everyone, but he realizes that if he tells it this way, you will probably decide that he is insane, so he instead chooses to put it diplomatically as “X is a huge problem”.
Taken together… you hired someone as an expert to provide an important information (and you congratulate yourself for doing it), but ultimately you ignored everything he said. And everyone felt happy in the process… your friend felt happy that the people making the important decisions were listening to him… your organization felt happy for having an ISO-certified process to listen to outside critics… except that ultimately nothing happened.
I am not saying that in a parallel reality where your friend instead wrote a Facebook post “we are all going to die” (and was ignored by everyone who matters) had a better outcome. But it had less self-deception by everyone involved.
Huh, this is a good quote.
What are “autists” supposed to do in a context like this?