To get the “orthogonality” part, I think the definition of the thesis also needs to include that increasing the intelligence of the agents does not cause interpretation of (some) goals to converge.
In particular, the dismissal of the concern that policy must include an absolutely perfect specification of the perfect goal does not deny that an agent could have a goal to maximize paperclip production, but rather asserts that the paperclip maximization goal may seed ASI adequately because a perfect intelligence pursuing it would behave the same as a perfect intelligence pursuing the perfect goal (although we imperfect intelligences do not realize this because we do not appreciate all the overlapping instrumental goals that both entail—for example, truly intelligent paperclip maximization may start with generating a maximally intelligent planner, and that may take so long that no actual paperclip get made).
Chris Santos-Lang
Karma: 10
Having documented epistemic consequences of realist metaphysics (https://doi.org/10.6084/m9.figshare.19204896.v1), it has been pointed-out to me that even non-realists might endorse them. That should be no surprise since non-realists are in community with realists, and sharing epistemic practices (like sharing language) makes community easier. No doubt, realists would likewise endorse non-realist practices, in return, if non-realism made any such practice necessary.
What makes this observation relevant to this article is the potential for non-realist positions to become irrelevant in practice because realist epistemic practices so deeply shape the practice of rationality. Community is an asset, so we can expect the best ASI to foster community, and for such community (including the ASI) to engage in realist epistemic practices. One might temporarily isolate into a “truth” selected to bring comfort, and one might even win political power by facilitating such comfort, but community division is unlikely to be sustainable. In other words, realism might inevitably end-up treated as true, even if false.