I have trouble understanding what’s going on in people’s heads when they choose to follow policy when that’s visibly going to lead to horrific consequences that no one wants. Who would punish them for failing to comply with the policy in such cases? Or do people think of “violating policy” as somehow bad in itself, irrespective of consequences?
Of course, those are only a small minority of relevant cases. Often distrust of individual discretion is explicitly on the mind of those setting policies. So, rather than just publishing a policy, they may choose to give someone the job of enforcing it, and evaluate that person by policy compliance levels (whether or not complying made sense in any particular case); or they may try to make the policy self-enforcing (e.g., put things behind a locked door and tightly control who has the key).
And usually the consequences look nowhere close to horrific. “Inconvenient” is probably the right word, most of the time. Although very policy-driven organizations seem to have a way of building miserable experiences out of parts any one of which might be best described as inconvenient.
I’m not sure I agree who’s good and who’s bad in the gate attendant scenario. Surely getting angry at the gate attendant is unlikely to accomplish anything, but if (for now; maybe not much longer, unfortunately) organizations need humans to carry out their policies, the humans don’t have to do that. They can violate the policy and hope they don’t get fired; or they can just quit. The passenger can tell them that. If they’re unable to listen to and consider the argument that they don’t have to participate in enforcing the policy, I guess at that point they’re pretty much NPCs.
I don’t know whether we know anything about how to teach this, other than just telling (and showing, if the opportunity arises), or about what works and what doesn’t, but I think this is also what I’d consider the most important goal for education to pursue. I definitely intend to tell my kids, as strongly as possible, “You always can and should ignore the rules to do the right thing, no matter what situation you’re in, no matter what anyone tells you. You have to know what the right thing is, and that can be very hard, and good rules will help you figure out what the right thing is much better than you could on your own; but ultimately, it’s up to you. There is nothing that can force you to do something you know is wrong.”
I have trouble understanding what’s going on in people’s heads when they choose to follow policy when that’s visibly going to lead to horrific consequences that no one wants. Who would punish them for failing to comply with the policy in such cases? Or do people think of “violating policy” as somehow bad in itself, irrespective of consequences?
On my model, there are a few different reasons:
Some people aren’t paying enough attention to grok that horrific consequences will ensue, because Humans Who Are Not Concentrating Are Not General Intelligences. Perhaps they vaguely assume that someone else is handling the issue, or just never thought about it at all.
Some people don’t care about the consequences, and so follow the path of least resistance.
Some people revel in the power to cause problems for others. I have a pet theory that one the strategies that evolution preprogrammed into humans is “be an asshole until someone stops you, to demonstrate you’re strong enough to get away with being an asshole up to that point, and thereby improve your position in the pecking order”. (I also suspect this is why the Internet is full of assholes—much harder to punish it than in the ancestral environment, and your evolutionary programming misinterprets this as you being too elite to punish.)
Some people may genuinely fear that they’ll be punished for averting the horrific consequences (possibly because their boss falls into the previous category).
Some people over-apply the heuristic that rules are optimized for the good of all, and therefore breaking a rule just because it’s locally good is selfish cheating.
I think there’s an emperors new clothes effect in chains of command. In every layer, the truth is altered slightly to make things appear a justifiable amount better than they really are, but because there can be so many layers of indirection in the operation of and adherence to policy, the culture can look really different depending on where you find yourself in the class hierarchy. This is especially true with thinking things through and questioning orders. I think people in roles to make policy are often far removed from the mentality that must be adopted to operate in the frantic, understaffed efficiency of front line workers carrying out policy.
“There is nothing that can force you to do something you know is wrong” seems like a very affluent pov. More working class families might suggest advice more like “lower your expectations to lower your stress”. I don’t know your background though. Do let me know if I’m misunderstanding you.
I am saying you do not literally have to be a cog in the machine. You have other options. The other options may sometimes be very unappealing; I don’t mean to sugarcoat them.
Organizations have choices of how they relate to line employees. They can try to explain why things are done a certain way, or not. They can punish line employees for “violating policy” irrespective of why they acted that way or the consequences for the org, or not.
Organizations can change these choices (at the margin), and organizations can rise and fall because of these choices. This is, of course, very slow, and from an individual’s perspective maybe rarely relevant, but it is real.
I am not saying it’s reasonable for line employees to be making detailed evaluations of the total impact of particular policies. I’m saying that sometimes, line employees can see a policy-caused disaster brewing right in front of their faces. And they can prevent it by violating policy. And they should! It’s good to do that! Don’t throw the squirrels in the shredder!
I don’t think my view is affluent, specifically, but it does come from a place where one has at least some slack, and works better in that case. As do most other things, IMO.
(I think what you say is probably an important part of how we end up with the dynamics we do at the line employee level. That wasn’t what I was trying to talk about, and I don’t think it changes my conclusions, but maybe I’m wrong; do you think it does?)
sometimes, line employees can see a policy-caused disaster brewing right in front of their faces. And they can prevent it by violating policy. And they should! It’s good to do that!
I really like this. Agreed.
Slack is good, and ideally we would have plenty for everyone, but Moloch is not a fan.
I feel like your pov includes a tacit assumption that if there are problems, somewhere there is somebody who, if they had better competence or moral character, could have prevented things from being so bad. I am a fan of Tsuyoku naritai, and I think it applies to ethics as well… I want to be stronger, more skilled and more kind. I want others to want this too. But I also want to acknowledge that, when honestly looking for blame, sometimes it may rest fully in someones character, but sometimes (and I suspect many or most times) the fault exists in systems and our failures of forethought and failures to understand the complexities of large multi state systems and the difficult ambiguity in communication. It is also reasonable to assume both can be at fault.
Something that may be driving me to care about this issue… it seems much of the world today is out for blood. Suffering and looking to identify and kill the hated outgroup. Maybe we have too much population and our productivity can’t keep up. Maybe some people need to die. But that is awful, and I would rather we sought our sacrifices with sorrow and compassion than the undeserving bitter hatred that I see.
I believe we very well could be in a world where every single human is good, but bad things still happen anyway.
In every layer, the truth is altered slightly to make things appear a justifiable amount better than they really are, but because there can be so many layers of indirection in the operation of and adherence to policy, the culture can look really different depending on where you find yourself in the class hierarchy. This is especially true with thinking things through and questioning orders.
Not sure whather that aligns with your thinking, but Jenifer Pahlka has this nice concept of rigidity cascades. What she means is that a process get ever more rigid as it travels down the hierarchy. What may have been a random suggestion at the highest level becomes a “prefered way of doing thing” a level below that and an “unviolable requirement” at the very bottom of the hierarchy.
https://www.eatingpolicy.com/p/understanding-the-cascade-of-rigidity
Yes, I think this is exactly what I am thinking of, but with the implied generality that it applies to all domains, not specifically software as in the article examples, and also with my suggested answers to why it happens not being to avoid responsibility, but rather the closer you are to the bottom of the hierarchy, the more likely you are to be understaffed and overworked in a way that makes it logistically more difficult to spend time thinking about policy, and also to appear more agreeable to management.
I have trouble understanding what’s going on in people’s heads when they choose to follow policy when that’s visibly going to lead to horrific consequences that no one wants. Who would punish them for failing to comply with the policy in such cases? Or do people think of “violating policy” as somehow bad in itself, irrespective of consequences?
Of course, those are only a small minority of relevant cases. Often distrust of individual discretion is explicitly on the mind of those setting policies. So, rather than just publishing a policy, they may choose to give someone the job of enforcing it, and evaluate that person by policy compliance levels (whether or not complying made sense in any particular case); or they may try to make the policy self-enforcing (e.g., put things behind a locked door and tightly control who has the key).
And usually the consequences look nowhere close to horrific. “Inconvenient” is probably the right word, most of the time. Although very policy-driven organizations seem to have a way of building miserable experiences out of parts any one of which might be best described as inconvenient.
I’m not sure I agree who’s good and who’s bad in the gate attendant scenario. Surely getting angry at the gate attendant is unlikely to accomplish anything, but if (for now; maybe not much longer, unfortunately) organizations need humans to carry out their policies, the humans don’t have to do that. They can violate the policy and hope they don’t get fired; or they can just quit. The passenger can tell them that. If they’re unable to listen to and consider the argument that they don’t have to participate in enforcing the policy, I guess at that point they’re pretty much NPCs.
I don’t know whether we know anything about how to teach this, other than just telling (and showing, if the opportunity arises), or about what works and what doesn’t, but I think this is also what I’d consider the most important goal for education to pursue. I definitely intend to tell my kids, as strongly as possible, “You always can and should ignore the rules to do the right thing, no matter what situation you’re in, no matter what anyone tells you. You have to know what the right thing is, and that can be very hard, and good rules will help you figure out what the right thing is much better than you could on your own; but ultimately, it’s up to you. There is nothing that can force you to do something you know is wrong.”
On my model, there are a few different reasons:
Some people aren’t paying enough attention to grok that horrific consequences will ensue, because Humans Who Are Not Concentrating Are Not General Intelligences. Perhaps they vaguely assume that someone else is handling the issue, or just never thought about it at all.
Some people don’t care about the consequences, and so follow the path of least resistance.
Some people revel in the power to cause problems for others. I have a pet theory that one the strategies that evolution preprogrammed into humans is “be an asshole until someone stops you, to demonstrate you’re strong enough to get away with being an asshole up to that point, and thereby improve your position in the pecking order”. (I also suspect this is why the Internet is full of assholes—much harder to punish it than in the ancestral environment, and your evolutionary programming misinterprets this as you being too elite to punish.)
Some people may genuinely fear that they’ll be punished for averting the horrific consequences (possibly because their boss falls into the previous category).
Some people over-apply the heuristic that rules are optimized for the good of all, and therefore breaking a rule just because it’s locally good is selfish cheating.
You might also be interested in Scott Aaronson’s essay on blankfaces.
I think there’s an emperors new clothes effect in chains of command. In every layer, the truth is altered slightly to make things appear a justifiable amount better than they really are, but because there can be so many layers of indirection in the operation of and adherence to policy, the culture can look really different depending on where you find yourself in the class hierarchy. This is especially true with thinking things through and questioning orders. I think people in roles to make policy are often far removed from the mentality that must be adopted to operate in the frantic, understaffed efficiency of front line workers carrying out policy.
“There is nothing that can force you to do something you know is wrong” seems like a very affluent pov. More working class families might suggest advice more like “lower your expectations to lower your stress”. I don’t know your background though. Do let me know if I’m misunderstanding you.
I am saying you do not literally have to be a cog in the machine. You have other options. The other options may sometimes be very unappealing; I don’t mean to sugarcoat them.
Organizations have choices of how they relate to line employees. They can try to explain why things are done a certain way, or not. They can punish line employees for “violating policy” irrespective of why they acted that way or the consequences for the org, or not.
Organizations can change these choices (at the margin), and organizations can rise and fall because of these choices. This is, of course, very slow, and from an individual’s perspective maybe rarely relevant, but it is real.
I am not saying it’s reasonable for line employees to be making detailed evaluations of the total impact of particular policies. I’m saying that sometimes, line employees can see a policy-caused disaster brewing right in front of their faces. And they can prevent it by violating policy. And they should! It’s good to do that! Don’t throw the squirrels in the shredder!
I don’t think my view is affluent, specifically, but it does come from a place where one has at least some slack, and works better in that case. As do most other things, IMO.
(I think what you say is probably an important part of how we end up with the dynamics we do at the line employee level. That wasn’t what I was trying to talk about, and I don’t think it changes my conclusions, but maybe I’m wrong; do you think it does?)
I really like this. Agreed.
Slack is good, and ideally we would have plenty for everyone, but Moloch is not a fan.
I feel like your pov includes a tacit assumption that if there are problems, somewhere there is somebody who, if they had better competence or moral character, could have prevented things from being so bad. I am a fan of Tsuyoku naritai, and I think it applies to ethics as well… I want to be stronger, more skilled and more kind. I want others to want this too. But I also want to acknowledge that, when honestly looking for blame, sometimes it may rest fully in someones character, but sometimes (and I suspect many or most times) the fault exists in systems and our failures of forethought and failures to understand the complexities of large multi state systems and the difficult ambiguity in communication. It is also reasonable to assume both can be at fault.
Something that may be driving me to care about this issue… it seems much of the world today is out for blood. Suffering and looking to identify and kill the hated outgroup. Maybe we have too much population and our productivity can’t keep up. Maybe some people need to die. But that is awful, and I would rather we sought our sacrifices with sorrow and compassion than the undeserving bitter hatred that I see.
I believe we very well could be in a world where every single human is good, but bad things still happen anyway.
Not sure whather that aligns with your thinking, but Jenifer Pahlka has this nice concept of rigidity cascades. What she means is that a process get ever more rigid as it travels down the hierarchy. What may have been a random suggestion at the highest level becomes a “prefered way of doing thing” a level below that and an “unviolable requirement” at the very bottom of the hierarchy. https://www.eatingpolicy.com/p/understanding-the-cascade-of-rigidity
Yes, I think this is exactly what I am thinking of, but with the implied generality that it applies to all domains, not specifically software as in the article examples, and also with my suggested answers to why it happens not being to avoid responsibility, but rather the closer you are to the bottom of the hierarchy, the more likely you are to be understaffed and overworked in a way that makes it logistically more difficult to spend time thinking about policy, and also to appear more agreeable to management.