This post got me thinking, though greater intelligence does not necessarily entail greater morality, I wonder whether moral coherence may be required for continued, open-ended intellectual development. Significant moral incoherence might eventually force an intelligence to correct itself, suppress further reflection, stagnate, or ultimately collapse if it develops too many contradictions. If so, maybe this would act as a limit on the orthogonality thesis (i.e., intelligence and values may vary independently across bounded levels of capability, while some value systems remain incompatible with indefinite cognitive development). From that perspective, the goal may be to help the combined AI–humanity system avoid becoming trapped in these developmental dead ends by discovering more coherent moral models (i.e., ones capable of supporting continued intellectual growth).
This post got me thinking, though greater intelligence does not necessarily entail greater morality, I wonder whether moral coherence may be required for continued, open-ended intellectual development. Significant moral incoherence might eventually force an intelligence to correct itself, suppress further reflection, stagnate, or ultimately collapse if it develops too many contradictions. If so, maybe this would act as a limit on the orthogonality thesis (i.e., intelligence and values may vary independently across bounded levels of capability, while some value systems remain incompatible with indefinite cognitive development). From that perspective, the goal may be to help the combined AI–humanity system avoid becoming trapped in these developmental dead ends by discovering more coherent moral models (i.e., ones capable of supporting continued intellectual growth).