Just thinking loudly about the boundaries of this effect...
Suppose I have a garden, and a robot that can pick apples. I instruct the robot to bring me the “nearest big apple” (where “big” is exactly defined as having a diameter at least X). Coincidentally, all apples in my garden are small, so the robot picks the nearest apple at neighbor’s garden and brings it to me, without saying anything.
When the neighbor notices it, he will be angry at me, but I can defend myself that I made an innocent mistake; I didn’t mean to steal apples. This may be a good excuse for the first time; but if the thing keeps happening, the excuse no longer works; it was based on me not knowing.
Also, the robot will not even try to be inconspicuous; it will climb straight over the fence, in the middle of a day, even when the neighbor is there and looking at it. If I try to avoid this, by giving instructions like “bring me the nearest apple, but make sure no one notices you”, I lose the plausible deniability. (I am assuming here that if my neighbor decides to sue me, the judge will see the robot’s programming.)
Now why specifically does the situation change when instead of a robot I use a slave? It seems to me the only difference is that I can (knowingly or not, or anything between that) achieve an outcome where the slave will steal for me, using his intelligence to avoid detection; and if caught anyway, I can afterwards claim that the slave decided to steal on his own will; not because of my commands (either ordering him to steal, or just failing to mention that theft is prohibited), but specifically and intentionally against them. That relieves of all blame.
Okay, so how specifically do I make a slave steal for me, without giving the explicit order? I have to give a command that is impossible (or highly unlikely) to accomplish otherwise. For example, my garden only contains small apples, and I command the slave to bring me a big apple; threatening to whip him if he fails. The only way to avoid whipping is to steal, but hey, I never mentioned that! Also, it is a common knowledge that (unlike the robot) the slave is supposed to know that stealing is wrong.
But this naive strategy fails if the slave checks my garden and then tells me “master, your garden only contains small apples; what am I supposed to do?” Saying “I don’t care, just bring the f-ing apple or feel my wrath” will achieve the purpose, but it puts a dent into my deniability. To do things properly, I must be able to claim afterwards that I believed that my garden contains big apples, and therefore I believed that my orders can be fulfilled legitimately. (Bonus points if afterwards, despite the slave’s testimony of the contrary, I can insist with indignation that my garden still contains big apples, i.e. the slave is just lying to save his ass.)
Therefore, I need to remove the communication channel where the slave could tell me that there are no big apples in my garden. A simple approach is to give the command, along with the threat, and then leave. The slave knows that he gets whipped if the apple is not on my table by sunset, and I am not there to communicate the fact that the order cannot be achieved legitimately. So he either takes the risk and waits for my return—betting on my sense of fairness, that I wouldn’t punish him for not having accomplished the impossible—or decides that it is relatively safer to simply steal the apple. This is better, but still too random. I could increase the probability of stealing by cultivating an image of a hot-tempered master who punishes firsts and asks questions later. But there are still ways this could fail (e.g. the slave could say afterwards “master, this was the last big apple in your garden, the remaining ones are all small”, which would remove my deniability for giving the same command the next day).
A more complex approach is to make it know that talking about small apples is a taboo, and all violations of this taboo will be punished. Now the slave will not dare mention that the apples in my garden are small, but now I need a plausible excuse for the taboo. Is perhaps my garden a source of my pride, and thus I take any criticism of my garden as a personal offense? That could work. I only need to establish the rule “I am proud of my garden and any kind of criticism will be severely punished” sufficiently long before I start ordering my slaves to steal; to avoid the impression that I established that rule exactly for that purpose.
Summary: a command that is impossible (or unlikely) to be accomplished legitimately; a threat of punishment; and proactively destroying the feedback channel under some pretense.
Now I’d like to give some advice on how to notice when you are in a similar situation—where the feedback channels are sabotaged, and perhaps it’s just a question of time when you receive the impossible command, and will have to use your own will to break the rules and take the responsibility for the decision—but actually, all situations with power differential are to smaller or greater degree like this. People usually can’t communicate with their superiors openly. (Not even when the boss says “actually, I prefer if you communicate with me openly”. Seriously, don’t. What that statement actually means is that talking about difficulties in communication to superiors is also a taboo. Generally, a company statement “X is true” is usually best translated as “saying ‘X is false’ will be punished”.)
A practical advice would perhaps be to notice the local taboos, and maneuver to avoid coming into contact with them. If a person X cannot be criticized, avoid being on the same team as X. If software Y cannot be criticized, avoid work that involves using software Y. If ideology Z cannot be criticized, avoid projects where Z applies most strongly. Yeah, I know, this is easier said than done, and it does not work reliably (e.g. you can choose a different team, and the next day X switches to that team, too).
Some context: In the past I had a job as a quality assurance inspector. I realized very soon after I started doing the job that a machine could easily do my job with less errors and for less then I was being paid so I wondered “Why do they pay for a human to do this job?” My conclusion was that if a machine makes a mistake as it is bound to do eventually they can’t really fire it or yell at it well as a human can be. A human can be blamed.
So I agree with you. In the future I can see robots doing all the jobs expect being the scape goats.
Self-driving cars have a similar problem. Even if the car would cause 100 times fewer accidents than a human driver, the problem is that when an accident happens, we need a human to blame.
How will we determine who goes to jail? Elon Musk? The poor programmer who wrote the piece of software that will be identified as having caused the bug? Or maybe someone like you, who “should have checked that the car is 100% safe”, even if everyone knows it is impossible. Most likely, it will be someone at the bottom of the corporate structure.
For now, as far as I know, the solution is that there must be a human driver in a self-driving car. In case of accident, that human will be blamed for not avoiding it by taking over the control.
But I suppose that moving the blame from the customer to some low-wage employee of the producer would be better for sales, so the legislation will likely change this way some day. We just need to find the proper scapegoat.
How will we determine who goes to jail? Elon Musk? The poor programmer who wrote the piece of software that will be identified as having caused the bug? Or maybe someone like you, who “should have checked that the car is 100% safe”, even if everyone knows it is impossible. Most likely, it will be someone at the bottom of the corporate structure.
It seems to me that the correct answer to your question is “no-one should go to jail”
Or more completely: In the absence of malice or extreme negligence there’s nothing criminal to punish at all and money damages should suffice. Given a 100x lower occurrence of accidents this should be insurable for ~1% the cost. The default answer is drivers remain financially responsible for damages (but insurance gets cheaper) and driver can’t be criminally negligent short of modifying/damaging the car in an obviously bad way (e.g. failing to fix a safety critical sensor in a reasonable amount of time that would have prevented the crash. Alternately, bypassing one or more safety features that could have prevented the crash). Car companies would be smart to lobby to keep it that way as letting every car accident become a product liability thing would be much more expensive.
This is an excellent analytical account of the underlying dynamics. It also VERY strongly resembles the series of blame-deflections described in Part II Chapter VII of Atlas Shrugged (the train-in-the-tunnel part), where this sort of information suppression ultimately backfires on the nominal beneficiary.
Just thinking loudly about the boundaries of this effect...
Suppose I have a garden, and a robot that can pick apples. I instruct the robot to bring me the “nearest big apple” (where “big” is exactly defined as having a diameter at least X). Coincidentally, all apples in my garden are small, so the robot picks the nearest apple at neighbor’s garden and brings it to me, without saying anything.
When the neighbor notices it, he will be angry at me, but I can defend myself that I made an innocent mistake; I didn’t mean to steal apples. This may be a good excuse for the first time; but if the thing keeps happening, the excuse no longer works; it was based on me not knowing.
Also, the robot will not even try to be inconspicuous; it will climb straight over the fence, in the middle of a day, even when the neighbor is there and looking at it. If I try to avoid this, by giving instructions like “bring me the nearest apple, but make sure no one notices you”, I lose the plausible deniability. (I am assuming here that if my neighbor decides to sue me, the judge will see the robot’s programming.)
Now why specifically does the situation change when instead of a robot I use a slave? It seems to me the only difference is that I can (knowingly or not, or anything between that) achieve an outcome where the slave will steal for me, using his intelligence to avoid detection; and if caught anyway, I can afterwards claim that the slave decided to steal on his own will; not because of my commands (either ordering him to steal, or just failing to mention that theft is prohibited), but specifically and intentionally against them. That relieves of all blame.
Okay, so how specifically do I make a slave steal for me, without giving the explicit order? I have to give a command that is impossible (or highly unlikely) to accomplish otherwise. For example, my garden only contains small apples, and I command the slave to bring me a big apple; threatening to whip him if he fails. The only way to avoid whipping is to steal, but hey, I never mentioned that! Also, it is a common knowledge that (unlike the robot) the slave is supposed to know that stealing is wrong.
But this naive strategy fails if the slave checks my garden and then tells me “master, your garden only contains small apples; what am I supposed to do?” Saying “I don’t care, just bring the f-ing apple or feel my wrath” will achieve the purpose, but it puts a dent into my deniability. To do things properly, I must be able to claim afterwards that I believed that my garden contains big apples, and therefore I believed that my orders can be fulfilled legitimately. (Bonus points if afterwards, despite the slave’s testimony of the contrary, I can insist with indignation that my garden still contains big apples, i.e. the slave is just lying to save his ass.)
Therefore, I need to remove the communication channel where the slave could tell me that there are no big apples in my garden. A simple approach is to give the command, along with the threat, and then leave. The slave knows that he gets whipped if the apple is not on my table by sunset, and I am not there to communicate the fact that the order cannot be achieved legitimately. So he either takes the risk and waits for my return—betting on my sense of fairness, that I wouldn’t punish him for not having accomplished the impossible—or decides that it is relatively safer to simply steal the apple. This is better, but still too random. I could increase the probability of stealing by cultivating an image of a hot-tempered master who punishes firsts and asks questions later. But there are still ways this could fail (e.g. the slave could say afterwards “master, this was the last big apple in your garden, the remaining ones are all small”, which would remove my deniability for giving the same command the next day).
A more complex approach is to make it know that talking about small apples is a taboo, and all violations of this taboo will be punished. Now the slave will not dare mention that the apples in my garden are small, but now I need a plausible excuse for the taboo. Is perhaps my garden a source of my pride, and thus I take any criticism of my garden as a personal offense? That could work. I only need to establish the rule “I am proud of my garden and any kind of criticism will be severely punished” sufficiently long before I start ordering my slaves to steal; to avoid the impression that I established that rule exactly for that purpose.
Summary: a command that is impossible (or unlikely) to be accomplished legitimately; a threat of punishment; and proactively destroying the feedback channel under some pretense.
Now I’d like to give some advice on how to notice when you are in a similar situation—where the feedback channels are sabotaged, and perhaps it’s just a question of time when you receive the impossible command, and will have to use your own will to break the rules and take the responsibility for the decision—but actually, all situations with power differential are to smaller or greater degree like this. People usually can’t communicate with their superiors openly. (Not even when the boss says “actually, I prefer if you communicate with me openly”. Seriously, don’t. What that statement actually means is that talking about difficulties in communication to superiors is also a taboo. Generally, a company statement “X is true” is usually best translated as “saying ‘X is false’ will be punished”.)
A practical advice would perhaps be to notice the local taboos, and maneuver to avoid coming into contact with them. If a person X cannot be criticized, avoid being on the same team as X. If software Y cannot be criticized, avoid work that involves using software Y. If ideology Z cannot be criticized, avoid projects where Z applies most strongly. Yeah, I know, this is easier said than done, and it does not work reliably (e.g. you can choose a different team, and the next day X switches to that team, too).
Some context: In the past I had a job as a quality assurance inspector. I realized very soon after I started doing the job that a machine could easily do my job with less errors and for less then I was being paid so I wondered “Why do they pay for a human to do this job?” My conclusion was that if a machine makes a mistake as it is bound to do eventually they can’t really fire it or yell at it well as a human can be. A human can be blamed.
So I agree with you. In the future I can see robots doing all the jobs expect being the scape goats.
Self-driving cars have a similar problem. Even if the car would cause 100 times fewer accidents than a human driver, the problem is that when an accident happens, we need a human to blame.
How will we determine who goes to jail? Elon Musk? The poor programmer who wrote the piece of software that will be identified as having caused the bug? Or maybe someone like you, who “should have checked that the car is 100% safe”, even if everyone knows it is impossible. Most likely, it will be someone at the bottom of the corporate structure.
For now, as far as I know, the solution is that there must be a human driver in a self-driving car. In case of accident, that human will be blamed for not avoiding it by taking over the control.
But I suppose that moving the blame from the customer to some low-wage employee of the producer would be better for sales, so the legislation will likely change this way some day. We just need to find the proper scapegoat.
It seems to me that the correct answer to your question is “no-one should go to jail”
Or more completely: In the absence of malice or extreme negligence there’s nothing criminal to punish at all and money damages should suffice. Given a 100x lower occurrence of accidents this should be insurable for ~1% the cost. The default answer is drivers remain financially responsible for damages (but insurance gets cheaper) and driver can’t be criminally negligent short of modifying/damaging the car in an obviously bad way (e.g. failing to fix a safety critical sensor in a reasonable amount of time that would have prevented the crash. Alternately, bypassing one or more safety features that could have prevented the crash). Car companies would be smart to lobby to keep it that way as letting every car accident become a product liability thing would be much more expensive.
This is an excellent analytical account of the underlying dynamics. It also VERY strongly resembles the series of blame-deflections described in Part II Chapter VII of Atlas Shrugged (the train-in-the-tunnel part), where this sort of information suppression ultimately backfires on the nominal beneficiary.