Has anyone attempted to prove the statement “Consciousness of a Turing machine is undecideable”? The proof (if it’s true) might look a lot like the proof that the halting problem is undecideable.
Your conjecture seems to follow from Rice’s theorem, assuming the personhood of a running computation is a property of the partial function its algorithm computes. Also, I think you can prove your conjecture by taking a certain proof that the Halting Problem is undecidable and replacing ‘halts’ with ‘is conscious’. I can track this down if you’re still interested.
But this doesn’t mess up Eliezer’s plans at all: you can have “nonhalting predicates” that output “doesn’t halt” or “I don’t know”, analogous to the nonperson predicates proposed here.
Your conjecture seems to follow from Rice’s theorem, assuming the personhood of a running computation is a property of the partial function its algorithm computes. Also, I think you can prove your conjecture by taking a certain proof that the Halting Problem is undecidable and replacing ‘halts’ with ‘is conscious’. I can track this down if you’re still interested.
But this doesn’t mess up Eliezer’s plans at all: you can have “nonhalting predicates” that output “doesn’t halt” or “I don’t know”, analogous to the nonperson predicates proposed here.