Eliezer: In FAI you’ve got to worry about a superintelligence searching huge amounts of argument-sequence space. My main thought for handling this sort of thing, consists of searching all ways and superposing them and considering the coherence or incoherence thereof as the extrapolated volition, rather than trying to map out one unique/optimal path.
But how to determine what kinds of modifications are allowed in the mix? Is hitting with an iron rod an argument? What is the coherence of superposition of extrapolations? Is lying dead and gradually decaying coherent?
Eliezer: In FAI you’ve got to worry about a superintelligence searching huge amounts of argument-sequence space. My main thought for handling this sort of thing, consists of searching all ways and superposing them and considering the coherence or incoherence thereof as the extrapolated volition, rather than trying to map out one unique/optimal path.
But how to determine what kinds of modifications are allowed in the mix? Is hitting with an iron rod an argument? What is the coherence of superposition of extrapolations? Is lying dead and gradually decaying coherent?