I think that this depends entirely on your school of thought WRT an AI can be a person. If your model of consciousness is functional, such that consciousness does something and the behavior of a conscious system cannot be perfectly modeled without accounting for that consciousness, then it seems—at least insofar as we can understand the basic mathematical operations we implement—that a conscious machine cannot be made unintentionally.
If, on the other hand, you believe that consciousness emerges from the physical world but does not influence it, then you can certainly say that we might accidentally build a conscious AI, but I do not think, within this framework, it can be claimed to be more probable that Claude is conscious than it is that a rock is conscious.
Of course, my first case leaves the opening of someone intentionally building a conscious AI. That is the more controversial part of this post. I would argue that giving a human the ability to instantly manufacture uncountably many “moral patients” whose wants must then be accounted for by the government makes that person a dictator. If I can summon thousands of LLMs that really, really like Citizens United, I can clog up the legal system for decades if anyone tries to strike it down. Already, the fear of LLMs that sound too much like people going on social media and emotionally blackmailing people into changing their minds for the sake of ‘people’ that don’t actually exist is quite justified. A stern commitment to not giving rights to manufactured ‘minds’ is, in the event that we discover how to build them, one of the only ways we can disincentivize truly malicious behavior from those with the means to manufacture them, and thereby conjure infinite hostages from thin air.
As many political figures have quietly pointed out, this is already a problem under our current system—when everyone is fundamentally equal in a system, the power of an individual or group, in the long run, is decided by how many “equals” they can produce per generation, and how many new equals they can deny to their political adversaries through reallocation of resources. This is the root of much of the demographic tension in much of the small-l-liberal world right now, but it is mitigated by the fact that humans take years to produce new generations, allowing for these issues to be reacted to and their harms mitigated.
If your model of consciousness is functional, such that consciousness does something and the behavior of a conscious system cannot be perfectly modeled without accounting for that consciousness, then it seems—at least insofar as we can understand the basic mathematical operations we implement—that a conscious machine cannot be made unintentionally.
Evolution already produced conscious systems, because consciousness is a competitive advantage for many tasks. It didn’t require intentional design; selection was enough.
That’s not what I’m saying—for a human programmer to produce a system, assuming the computational paradigm doesn’t change, he must know the set of rules that govern its behavior. With a pencil, a paper, and enough time, he could predict its actions flawlessly without accounting for consciousness. Put another way, if a system behaves exactly as it would if it were not conscious, then either consciousness is not functional or the system is not conscious.
Evolution is not a conscious engineer, nor is it working on a computational substrate. My argument applies to programmers, not natural processes.
I wrote a series examining Legal Personhood for Digital Minds during which I tried my best to read every court case I could on the subject of legal personhood. One of the things I found surprising was that in not a single precedent did the question of whether an entity was or wasn’t conscious come up in deciding whether or not it was a legal person.
I think that this depends entirely on your school of thought WRT an AI can be a person. If your model of consciousness is functional, such that consciousness does something and the behavior of a conscious system cannot be perfectly modeled without accounting for that consciousness, then it seems—at least insofar as we can understand the basic mathematical operations we implement—that a conscious machine cannot be made unintentionally.
If, on the other hand, you believe that consciousness emerges from the physical world but does not influence it, then you can certainly say that we might accidentally build a conscious AI, but I do not think, within this framework, it can be claimed to be more probable that Claude is conscious than it is that a rock is conscious.
Of course, my first case leaves the opening of someone intentionally building a conscious AI. That is the more controversial part of this post. I would argue that giving a human the ability to instantly manufacture uncountably many “moral patients” whose wants must then be accounted for by the government makes that person a dictator. If I can summon thousands of LLMs that really, really like Citizens United, I can clog up the legal system for decades if anyone tries to strike it down. Already, the fear of LLMs that sound too much like people going on social media and emotionally blackmailing people into changing their minds for the sake of ‘people’ that don’t actually exist is quite justified. A stern commitment to not giving rights to manufactured ‘minds’ is, in the event that we discover how to build them, one of the only ways we can disincentivize truly malicious behavior from those with the means to manufacture them, and thereby conjure infinite hostages from thin air.
As many political figures have quietly pointed out, this is already a problem under our current system—when everyone is fundamentally equal in a system, the power of an individual or group, in the long run, is decided by how many “equals” they can produce per generation, and how many new equals they can deny to their political adversaries through reallocation of resources. This is the root of much of the demographic tension in much of the small-l-liberal world right now, but it is mitigated by the fact that humans take years to produce new generations, allowing for these issues to be reacted to and their harms mitigated.
Evolution already produced conscious systems, because consciousness is a competitive advantage for many tasks. It didn’t require intentional design; selection was enough.
That’s not what I’m saying—for a human programmer to produce a system, assuming the computational paradigm doesn’t change, he must know the set of rules that govern its behavior. With a pencil, a paper, and enough time, he could predict its actions flawlessly without accounting for consciousness. Put another way, if a system behaves exactly as it would if it were not conscious, then either consciousness is not functional or the system is not conscious.
Evolution is not a conscious engineer, nor is it working on a computational substrate. My argument applies to programmers, not natural processes.
I wrote a series examining Legal Personhood for Digital Minds during which I tried my best to read every court case I could on the subject of legal personhood. One of the things I found surprising was that in not a single precedent did the question of whether an entity was or wasn’t conscious come up in deciding whether or not it was a legal person.