The great majority of those who judge increases to intelligence to be worse than the status quo would also judge decreases to be worse than the status quo. But this puts them in the rather odd position of maintaining that the net value for society provided by our current level of intelligence is at a local optimum, with small changes in either direction producing something worse. We can then ask for an explanation of why this should be thought to be so. If no sufficient reason is provided, our suspicion that the original judgment was influenced by status quo bias is corroborated.
[. . .]
The rationale of the Reversal Test is simple: if a continuous parameter admits of a wide range of possible values, only a tiny subset of which can be local optima, then it is prima facie implausible that the actual value of that parameter should just happen to be at one of these rare local optima [. . .] the burden of proof shifts to those who maintain that some actual parameter is at such a local optimum: they need to provide some good reason for supposing that it is so.
Obviously, the Reversal Test does not show that preferring the status quo is always unjustified. In many cases, it is possible to meet the challenge posed by the Reversal Test [. . .] Let us examine some of the possible ways [. . .]
The potential harms and benefits of intelligence depend partly on the nature of the system they exist in. Shift a system away from equilibrium, and harm will tend to predominate in the consequences until the system adapts. Adaptation takes time, and sometimes a great deal of learning.
We don’t need to believe that we’re at some kind of ultimate optimum to doubt whether a sudden change would be beneficial… as the arguments you mention suggest.
If one variable changes, the adaptations to the old value will no have the old effect—and they’re far more likely to be harmful than beneficial.
Imagine that people became more truthful overnight. All of the societal factors that relied on people being deceptive will be thrown out of balance. Benefits predicated upon a certain level of deception will not longer occur, and rules designed to induce people to be truthful in certain situations (like court proceedings) may end up causing more harm than good.
Imagine if people actually took the oath “the truth, the whole truth, and nothing but the truth” seriously all of a sudden. Chaos, paralysis, and malfunctioning would be the immediate effects.
On second thought, no, it’s not always true. Both the improvement after the equilibrium is reached again, and change involving the confusion just after the change, are dependent on the extent of the change (and there need not be a sudden change, the whole process can be performed by shifting the state of adaptation). While state after-change is worse than state after-calming-down, both can be better than the initial state, or both worse than initial state, as well as on the different sides of the initial state.
Related: The Reversal Test: Eliminating Status Quo Biases in Applied Ethics, by Nick Bostrom and Toby Ord.
The potential harms and benefits of intelligence depend partly on the nature of the system they exist in. Shift a system away from equilibrium, and harm will tend to predominate in the consequences until the system adapts. Adaptation takes time, and sometimes a great deal of learning.
We don’t need to believe that we’re at some kind of ultimate optimum to doubt whether a sudden change would be beneficial… as the arguments you mention suggest.
In what you describe, the fact that the system is adapted in the current context is the reason the current context is a local optimum.
Yes, that’s always true.
If one variable changes, the adaptations to the old value will no have the old effect—and they’re far more likely to be harmful than beneficial.
Imagine that people became more truthful overnight. All of the societal factors that relied on people being deceptive will be thrown out of balance. Benefits predicated upon a certain level of deception will not longer occur, and rules designed to induce people to be truthful in certain situations (like court proceedings) may end up causing more harm than good.
Imagine if people actually took the oath “the truth, the whole truth, and nothing but the truth” seriously all of a sudden. Chaos, paralysis, and malfunctioning would be the immediate effects.
On second thought, no, it’s not always true. Both the improvement after the equilibrium is reached again, and change involving the confusion just after the change, are dependent on the extent of the change (and there need not be a sudden change, the whole process can be performed by shifting the state of adaptation). While state after-change is worse than state after-calming-down, both can be better than the initial state, or both worse than initial state, as well as on the different sides of the initial state.
It’s highly unlikely that a shift away from equilibrium will be directly beneficial—but you’re right, it is possible.
I retract my earlier statement and qualify it.
Thanks, I created an article on the wiki, citing the paper and using your quote:
Reversal test.