The orthogonality question is an engineering question
People usually think about the orthogonality question (“Is the orthogonality thesis true?”) as a philosophical question. The usual way of approaching the orthogonality question is by taking a starting point of “assume an AGI exists” and then reasoning about what goals the AGI would have. But one can flip the usual starting point around and ask, for a specific goal, “is it realistically achievable to create a general intelligence that has this goal?” This reframing turns the orthogonality question into an engineering question that has more direct practical relevance than the philosophical version. The engineering version is a question about the types of results an AI developer can expect from different engineering decisions, rather than speculation about an idealized AGI; it’s grounded in what’s realistically achievable instead of what might be theoretically possible.
Instances of the engineering version of the orthogonality question also open the broader orthogonality question up to empirical testing. And so far, the empirical evidence we’ve received has pointed toward the answer “no.” Ever since the early days of reinforcement learning, researchers have been creating models with narrow goals, and so far, none of those systems has shown full generalization in the type of intelligence it’s developed. Protein-folding models only fold proteins; chess engines don’t model their environments outside the confines of the 64 squares. Language prediction has generalized further than most other training objectives, but language models still perform poorly at non-linguistic tasks (understanding images, acting within physical environments) and have jagged capabilities even within the set of language-based problems. Each new failure to create general intelligence from a narrow training objective is (usually weak) empirical evidence that narrow training signals are too impoverished to let a model develop highly general capabilities. Maybe general intelligence from a narrow goal would be possible with truly gargantuan amounts of compute, but recall that the engineering version of the orthogonality question is about what’s practically achievable.
The orthogonality question is an engineering question
People usually think about the orthogonality question (“Is the orthogonality thesis true?”) as a philosophical question. The usual way of approaching the orthogonality question is by taking a starting point of “assume an AGI exists” and then reasoning about what goals the AGI would have. But one can flip the usual starting point around and ask, for a specific goal, “is it realistically achievable to create a general intelligence that has this goal?” This reframing turns the orthogonality question into an engineering question that has more direct practical relevance than the philosophical version. The engineering version is a question about the types of results an AI developer can expect from different engineering decisions, rather than speculation about an idealized AGI; it’s grounded in what’s realistically achievable instead of what might be theoretically possible.
Instances of the engineering version of the orthogonality question also open the broader orthogonality question up to empirical testing. And so far, the empirical evidence we’ve received has pointed toward the answer “no.” Ever since the early days of reinforcement learning, researchers have been creating models with narrow goals, and so far, none of those systems has shown full generalization in the type of intelligence it’s developed. Protein-folding models only fold proteins; chess engines don’t model their environments outside the confines of the 64 squares. Language prediction has generalized further than most other training objectives, but language models still perform poorly at non-linguistic tasks (understanding images, acting within physical environments) and have jagged capabilities even within the set of language-based problems. Each new failure to create general intelligence from a narrow training objective is (usually weak) empirical evidence that narrow training signals are too impoverished to let a model develop highly general capabilities. Maybe general intelligence from a narrow goal would be possible with truly gargantuan amounts of compute, but recall that the engineering version of the orthogonality question is about what’s practically achievable.