If your model of consciousness is functional, such that consciousness does something and the behavior of a conscious system cannot be perfectly modeled without accounting for that consciousness, then it seems—at least insofar as we can understand the basic mathematical operations we implement—that a conscious machine cannot be made unintentionally.
Evolution already produced conscious systems, because consciousness is a competitive advantage for many tasks. It didn’t require intentional design; selection was enough.
That’s not what I’m saying—for a human programmer to produce a system, assuming the computational paradigm doesn’t change, he must know the set of rules that govern its behavior. With a pencil, a paper, and enough time, he could predict its actions flawlessly without accounting for consciousness. Put another way, if a system behaves exactly as it would if it were not conscious, then either consciousness is not functional or the system is not conscious.
Evolution is not a conscious engineer, nor is it working on a computational substrate. My argument applies to programmers, not natural processes.
Evolution already produced conscious systems, because consciousness is a competitive advantage for many tasks. It didn’t require intentional design; selection was enough.
That’s not what I’m saying—for a human programmer to produce a system, assuming the computational paradigm doesn’t change, he must know the set of rules that govern its behavior. With a pencil, a paper, and enough time, he could predict its actions flawlessly without accounting for consciousness. Put another way, if a system behaves exactly as it would if it were not conscious, then either consciousness is not functional or the system is not conscious.
Evolution is not a conscious engineer, nor is it working on a computational substrate. My argument applies to programmers, not natural processes.