High-intelligence agents can exist having more or less any final goals (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent’s intelligence).
Old post:
I was just working through my own thoughts on the Orthogonality thesis and did a search on LW on existing material and found this essay. I had pretty much the same thoughts on intelligence limiting goal complexity, so yay!
Additional thought I had: Learning/intelligence-boosting motivations/goals are positively correlated with intelligence. Thus, given any amount of time, an AI with intelligence-boosting motivations will become smarter than those do not have that motivation.
It is true that instrumental convergence should lead any sufficiently smart AI to also pursue intelligence-boosting (cognitive enhancement) but:
At low levels of intelligence, AI might fail at instrumental convergence strategies.
At high levels of intelligence, AI that is not intelligence-boosting will spend some non-zero amount of resources on its actual other goals and thus be less intelligent than an intelligence-boosting AI (assuming parallel universes, and thus no direct competition).
I’m not sure how to integrate this insight in to the orthogonality thesis. It implies that:
“At higher intelligence levels, intelligence-boosting motivations are more likely than other motivations” thus creating a probability distribution across the intelligence-goal space that I’m not sure how to represent. Thoughts?
EDIT: Both are points are moot using Stuart Armstrong’s narrower definition of the Orthogonality thesis that he argues in General purpose intelligence: arguing the Orthogonality thesis:
Old post:
I was just working through my own thoughts on the Orthogonality thesis and did a search on LW on existing material and found this essay. I had pretty much the same thoughts on intelligence limiting goal complexity, so yay!
Additional thought I had: Learning/intelligence-boosting motivations/goals are positively correlated with intelligence. Thus, given any amount of time, an AI with intelligence-boosting motivations will become smarter than those do not have that motivation.
It is true that instrumental convergence should lead any sufficiently smart AI to also pursue intelligence-boosting (cognitive enhancement) but:
At low levels of intelligence, AI might fail at instrumental convergence strategies.
At high levels of intelligence, AI that is not intelligence-boosting will spend some non-zero amount of resources on its actual other goals and thus be less intelligent than an intelligence-boosting AI (assuming parallel universes, and thus no direct competition).
I’m not sure how to integrate this insight in to the orthogonality thesis. It implies that:
“At higher intelligence levels, intelligence-boosting motivations are more likely than other motivations” thus creating a probability distribution across the intelligence-goal space that I’m not sure how to represent. Thoughts?