I’m hoping to complement this with a paper laying out the positive arguments in favour of the thesis. So I’m asking you for your strongest arguments for (or against) the orthogonality thesis.
Any kind of agent could—in principle—be engineered.
However, some sorts of agent are more likely to evolve than others—and it is this case that actually matters to us.
For example, intelligent machines are likely to coevolve in a symbiosis with humans—during which they will pick up some of our values. In this case, intelligence and values will be powerfully linked—since stupid machines will fail to absorb so many of our values—as we have seen, for example, with the evolution of cars.
So: The Orthogonality Thesis:
Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.
...is true[*] - but the “in principle” renders it kind-of irrelevant to the case that we actually care about.
* Unless the wirehead / pornography problems turn out to actually be serious issues.
I have doubts that it is even true “in principle” unless the goals are hard-wired in and unmodifiable by the intelligence. Do you really think that someone would agree to be OCD or schizophrenic if they had a choice? For higher levels of intelligence, I would think they would be even more discriminating as to goal-states they would accept.
As for the argument by thelittledoctor, the evil genius dictator model is broken even for highly intelligent humans, much less super-intelligences. Those “intelligent” demagogues are rarely, if ever, more than 2 standard deviations above average human intelligence, that definitely doesn’t count as “highly intelligent” as far as I’m concerned.
It seems irrelevant whether the AI is quote-unquote “highly intelligent” as long as it’s clever enough to take over a country and kill several million people.
The usual argument is that we are likely to be able to build machines that won’t want to modify their goals.
IMO, the more pressing issue with something like OCD is that it might interfere with intelligence tests—in which case you could argue that an OCD superintelligent machine is not really intelligent—since it is using its intelligence to screw itself.
This seems to be a corner case to me. The intended point is more that you could engineer an evil genius, or an autistic mind child.
Any kind of agent could—in principle—be engineered.
However, some sorts of agent are more likely to evolve than others—and it is this case that actually matters to us.
For example, intelligent machines are likely to coevolve in a symbiosis with humans—during which they will pick up some of our values. In this case, intelligence and values will be powerfully linked—since stupid machines will fail to absorb so many of our values—as we have seen, for example, with the evolution of cars.
So: The Orthogonality Thesis:
...is true[*] - but the “in principle” renders it kind-of irrelevant to the case that we actually care about.
* Unless the wirehead / pornography problems turn out to actually be serious issues.
I have doubts that it is even true “in principle” unless the goals are hard-wired in and unmodifiable by the intelligence. Do you really think that someone would agree to be OCD or schizophrenic if they had a choice? For higher levels of intelligence, I would think they would be even more discriminating as to goal-states they would accept.
As for the argument by thelittledoctor, the evil genius dictator model is broken even for highly intelligent humans, much less super-intelligences. Those “intelligent” demagogues are rarely, if ever, more than 2 standard deviations above average human intelligence, that definitely doesn’t count as “highly intelligent” as far as I’m concerned.
It seems irrelevant whether the AI is quote-unquote “highly intelligent” as long as it’s clever enough to take over a country and kill several million people.
The usual argument is that we are likely to be able to build machines that won’t want to modify their goals.
IMO, the more pressing issue with something like OCD is that it might interfere with intelligence tests—in which case you could argue that an OCD superintelligent machine is not really intelligent—since it is using its intelligence to screw itself.
This seems to be a corner case to me. The intended point is more that you could engineer an evil genius, or an autistic mind child.