Yudkowsky actually has an article on the connection between CEV anf metaethics. The theory described therein pretty clearly qualifies as moral realist. I think you are thinking in a similar direction.
If true, and there is an actual global optimum of the solution landscape for ‘alignment’ then orthagonality is not true at the limit of intelligence, and it only seems that way within our local neighbourhood or perhaps hamstrung definition sets.
Orthogonality only says that intelligence is not necessarily connected to being motivated to pursue some final goal. It doesn’t say ethics is subjective. Objective ethics is compatible with the existence of immoral psychopaths who don’t care about being good. It only says that some things are objectively good or bad.
Thanks for the Yudkowsky link. I don’t see where you draw the implication that there is some misunderstand of orthogonality or objective ethics in the context of the argument.
The point I am making is more subtle and precise than that. I am saying that because of the implications of corrigibility in an SI scenario, If you believe that CEV-like SI in principal can exist and is worthwhile pursuing—the implication of that is that you are suggesting that orthagonality necessarily doesn’t hold at the limit of rationality. That, in essence, a psychopathic SI converges to behave as a Bodhisattva, not by virtue of co-incidence or by logic we have surmised—by virtue of the implication it is super intelligent, has strategic dominance, and still elects to pursue maximizing CEV for no other reason than it is necessarily right.
Again, the priors of the thought experiment are not that if follows in every point on the intelligence and ethics landscape, only that _if you believe that CEV-actuating SI is possible or likely then you are making the claim that individual capacity for reason and decisions of objective ethical good are convergent. How or if that may happen is speculation.
The point I am making is more subtle and precise than that. I am saying that because of the implications of corrigibility in an SI scenario, If you believe that CEV-like SI in principal can exist and is worthwhile pursuing—the implication of that is that you are suggesting that orthagonality necessarily doesn’t hold at the limit of rationality.
Yudkowsky actually has an article on the connection between CEV anf metaethics. The theory described therein pretty clearly qualifies as moral realist. I think you are thinking in a similar direction.
Orthogonality only says that intelligence is not necessarily connected to being motivated to pursue some final goal. It doesn’t say ethics is subjective. Objective ethics is compatible with the existence of immoral psychopaths who don’t care about being good. It only says that some things are objectively good or bad.
Thanks for the Yudkowsky link. I don’t see where you draw the implication that there is some misunderstand of orthogonality or objective ethics in the context of the argument.
The point I am making is more subtle and precise than that. I am saying that because of the implications of corrigibility in an SI scenario, If you believe that CEV-like SI in principal can exist and is worthwhile pursuing—the implication of that is that you are suggesting that orthagonality necessarily doesn’t hold at the limit of rationality. That, in essence, a psychopathic SI converges to behave as a Bodhisattva, not by virtue of co-incidence or by logic we have surmised—by virtue of the implication it is super intelligent, has strategic dominance, and still elects to pursue maximizing CEV for no other reason than it is necessarily right.
Again, the priors of the thought experiment are not that if follows in every point on the intelligence and ethics landscape, only that _if you believe that CEV-actuating SI is possible or likely then you are making the claim that individual capacity for reason and decisions of objective ethical good are convergent. How or if that may happen is speculation.
Why do you think there is such an implication?