The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal, that is, its final goals and intelligence levels can vary independently of each other. This is in contrast to the belief that, because of their intelligence, AIs will all converge to a common goal.
The thesis was originally defined by Nick Bostrom in the paper “Superintelligent Will”, (along with the instrumental convergence thesis). For his purposes, Bostrom defines intelligence to be instrumental rationality.
Related: Complexity of Value, Decision Theory, General Intelligence, Utility Functions
Defense of the thesis
It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs. Stuart Armstrong writes that,
One reason many researchers assume superintelligent agents to converge to the same goals may be because most humans have similar values. Furthermore, many philosophies hold that there is a rationally correct morality, which implies that a sufficiently rational AI will acquire this morality and begin to act according to it. Armstrong points out that for formalizations of AI such as AIXI and Gödel machines, the thesis is known to be true. Furthermore, if the thesis was false, then Oracle AIs would be impossible to build, and all sufficiently intelligent AIs would be impossible to control.
Pathological Cases
There are some pairings of intelligence and goals which cannot exist. For instance, an AI may have the goal of using as little resources as possible, or simply of being as unintelligent as possible. These goals will inherently limit the degree of intelligence of the AI.
See Also
External links
Definition of the orthogonality thesis from Bostrom’s Superintelligent Will
Critique of the thesis by John Danaher
Superintelligent Will paper by Nick Bostrom
From the old discussion page:
Talk:Orthogonality thesis
Quality Concerns
There is no reference for Stuart Armstrong’s requirements. I guess they come from here.
I know roughly what the orthogonality thesis is. But if I’d only read the wiki page, it wouldn’t make sense to me. – »Well, we don’t share the opinion of some people that the goals of increasingly intelligent agents converge. So we put up a thesis which claims that intelligence and goals vary freely. Suppose there’s one goal system that fulfils Armstrong’s requirements. This would refute the orthogonality thesis, even if most intelligences converged on one or two other goal systems.« I don’t mean to say that the orthogonality thesis itself doesn’t make sense. I mean that the wiki page doesn’t provide enough information to enable people to understand that it makes sense.
Instead of repairing the above shortcomings, I propose referring people to the corresponding Arbital page. --Rmoehn (talk) 14:47, 31 May 2016 (AEST)