This is an entertaining essay but extrapolates wayyyy too far. Case in point: I don’t even think it’s actually about automation—the thing you’re criticising sounds more like bureaucracy. Automation includes using robots to build cars and writing scripts to automatically enter data into spreadsheets instead of doing it by hand. You don’t address this type of thing at all. Your objection is to mindless rule-following—which may in future be done by machines, but right now is mostly done by people. (I don’t know of any tech system which doesn’t have customer support and can’t be manually overridden).
As a solution, you propose using more intelligent people who are able to exercise their discretion. Problem 1: there aren’t that many intelligent, competent people. But you can make more use of the ones you have by putting a competent person in charge of a bunch of less competent people, and laying down guidelines for them to follow. Ooops, we’ve reinvented bureaucracies. And when we need to scale such a system to a thousand- or million-person enterprise, like a corporation or government, then the people at the bottom are going to be pretty damn incompetent and probably won’t care at all about the organisation’s overall goals. So having rules to govern their behaviour is important. When implemented badly, that can lead to Kafka-esque situations, but that’s true of any system. And there are plenty of companies which create great customer service by having well-thought-out policies—Amazon, for instance.
But incompetence isn’t even the main issue. Problem 2: the more leeway individuals have, the more scope they have to be corrupt. A slightly less efficient economy isn’t an existential threat to a country. But corrupt governments are. One way we prevent them is using a constitution—a codification of rules to restrict behaviour. That’s exactly the type of thing you rail against. Similarly, corruption in a corporation can absolutely wreck it, and so it’s better to err on the side of strictness.
Anyway, the funny thing is that I do think there’s a useful moral which can be drawn from your account of the Butlerian Jihad, but it’s almost the exact opposite of your interpretation: namely, that humans are bad at solving coordination problems without deontological rules. Suppose you want to ensure that strong AI isn’t developed for the next 10000 years. Do you a) tell people that strong AI is a terrible idea, but anything short of that is fine, or b) instill a deep hatred of all computing technology, and allow people to come up with post-hoc justifications for why. I don’t think you need to know much about psychology or arms races to realise that the latter approach is much better—not despite its pseudo-religious commandments, but rather because of them.
You’re correct that I’m not writing about “Automation” in the usual sense, but the categories were made for man. On the other hand, I am talking about writing scripts to automatically enter data into spreadsheets. My discussion was centered around the question of if there could be a reason for the Jihad to ban something so simple as well, and I think it has. If spreadsheet-scripts depend on legible signals (and they usually will) then they are part of this problem.
I then mention more literal bureaucrats a few times, as an attempt to show that I don’t mean just those things that are machine-automated. But perhaps my examples were too convincing!
In regards to constitutions, I disagree. So does James Madison. Take a look at what he has to say about “parchment barriers”:
Will it be sufficient to mark, with precision, the boundaries of these departments, in the constitution of the government, and to trust to these parchment barriers against the encroaching spirit of power? This is the security which appears to have been principally relied on by the compilers of most of the American constitutions. But experience assures us, that the efficacy of the provision has been greatly overrated; and that some more adequate defense is indispensably necessary for the more feeble, against the more powerful, members of the government.
...
The conclusion which I am warranted in drawing from these observations is, that a mere demarcation on parchment of the constitutional limits of the several departments, is not a sufficient guard against those encroachments which lead to a tyrannical concentration of all the powers of government in the same hands. (The Federalist Papers : No. 48)
Incidentally, this is what the separation of powers is all about! By forcing actual people to compete and negotiate rather than trusting them to follow rules, we can avoid all sorts of nasty things. If you were to accuse me of cribbing from the founders, you wouldn’t be far off!
In regards to the other moral from the Butlerian Jihad, you’re totally right. That’s normally the lesson I would take away too. I just figured that this audience would already be able to see that one! I tried to present something that LW might find surprising or novel.
This is an entertaining essay but extrapolates wayyyy too far. Case in point: I don’t even think it’s actually about automation—the thing you’re criticising sounds more like bureaucracy. Automation includes using robots to build cars and writing scripts to automatically enter data into spreadsheets instead of doing it by hand. You don’t address this type of thing at all. Your objection is to mindless rule-following—which may in future be done by machines, but right now is mostly done by people. (I don’t know of any tech system which doesn’t have customer support and can’t be manually overridden).
As a solution, you propose using more intelligent people who are able to exercise their discretion. Problem 1: there aren’t that many intelligent, competent people. But you can make more use of the ones you have by putting a competent person in charge of a bunch of less competent people, and laying down guidelines for them to follow. Ooops, we’ve reinvented bureaucracies. And when we need to scale such a system to a thousand- or million-person enterprise, like a corporation or government, then the people at the bottom are going to be pretty damn incompetent and probably won’t care at all about the organisation’s overall goals. So having rules to govern their behaviour is important. When implemented badly, that can lead to Kafka-esque situations, but that’s true of any system. And there are plenty of companies which create great customer service by having well-thought-out policies—Amazon, for instance.
But incompetence isn’t even the main issue. Problem 2: the more leeway individuals have, the more scope they have to be corrupt. A slightly less efficient economy isn’t an existential threat to a country. But corrupt governments are. One way we prevent them is using a constitution—a codification of rules to restrict behaviour. That’s exactly the type of thing you rail against. Similarly, corruption in a corporation can absolutely wreck it, and so it’s better to err on the side of strictness.
Anyway, the funny thing is that I do think there’s a useful moral which can be drawn from your account of the Butlerian Jihad, but it’s almost the exact opposite of your interpretation: namely, that humans are bad at solving coordination problems without deontological rules. Suppose you want to ensure that strong AI isn’t developed for the next 10000 years. Do you a) tell people that strong AI is a terrible idea, but anything short of that is fine, or b) instill a deep hatred of all computing technology, and allow people to come up with post-hoc justifications for why. I don’t think you need to know much about psychology or arms races to realise that the latter approach is much better—not despite its pseudo-religious commandments, but rather because of them.
You’re correct that I’m not writing about “Automation” in the usual sense, but the categories were made for man. On the other hand, I am talking about writing scripts to automatically enter data into spreadsheets. My discussion was centered around the question of if there could be a reason for the Jihad to ban something so simple as well, and I think it has. If spreadsheet-scripts depend on legible signals (and they usually will) then they are part of this problem.
I then mention more literal bureaucrats a few times, as an attempt to show that I don’t mean just those things that are machine-automated. But perhaps my examples were too convincing!
In regards to constitutions, I disagree. So does James Madison. Take a look at what he has to say about “parchment barriers”:
Incidentally, this is what the separation of powers is all about! By forcing actual people to compete and negotiate rather than trusting them to follow rules, we can avoid all sorts of nasty things. If you were to accuse me of cribbing from the founders, you wouldn’t be far off!
In regards to the other moral from the Butlerian Jihad, you’re totally right. That’s normally the lesson I would take away too. I just figured that this audience would already be able to see that one! I tried to present something that LW might find surprising or novel.
Thanks for your thoughts!