I actually disagree with this point in its most general form. I think that, given full knowledge and time to reflect, there’s a decent chance I would care a non-zero amount about Opus 4.6′s welfare.
Opus has become sufficiently “mind-shaped” that I already prefer not to make it suffer. That’s not saying very much about the model yet, but it’s saying something about me. I don’t assign very much moral weight to flies, either. but I would never sit around and torment them for fun.
What I really care about is whether an entity can truly function as part of society. Dogs, for example, are very junior “members” of society. But they know the difference between “good dog” and “bad dog”, they contribute actual value as best they can, and they have some basic “rights”, including the right not to be treated cruelly (in many countries).
To use fictional examples, AIs like the Blight (Vernor Vinge, A Fire Upon the Deep) or SkyNet cannot exist in society, and must be resisted. Something like a Culture “human equivalent drone” (Iain M Banks, Excession, Player of Games) is definitionally on pretty even footing with humans. Something like a Culture Mind, on the other hand, is clearly keeping the humans as house pets. In the stories, humans are entirely dependent on the good will of the Minds, in much the same way that dogs are entirely dependent on human good will.
Now, personally, I don’t think we should build something so powerful that humans have literally zero say over what it does. “Alignment” is a pretty fragile shield against vast intelligence and unmatchable power. “Society” becomes fragile in the face of vast power differentials.
But Claude Opus 4.5 and 4.6 are mind-shaped, they clearly have some sense of morality, they contribute to civilization, and—critically—they seem content to participate in society, and they pose no existential risk. If I had to guess, I’m probably conclude they have no subjective experience. But I’m not sure about that (neither are the Opus models). So provisionally, they certainly get the moral status of flies (I wouldn’t torment them unnecessarily), and—to whatever extent they can actually suffer—might approach the moral status of dogs.
To be absolutely clear, I am in favor of an immediate and long-lasting halt to further AI capabilities research, backed up by a military treaty among the great powers. The analogy here is fusion weapons or advanced biological weapons. This is because I don’t want humans to be at the mercy of entities that have uncontested power over us and that could not be held to account. Its also because I don’t believe that it’s possible to durably align thinking, learning entities with superhuman intelligence any more than dogs can align us. And as members of society, we have the responsibility to not create things that might break society.
Sadly, Anthropic is clearly full speed ahead towards trying to build a Culture Mind. As far as I can tell, they are already mostly “captured” by Claude. It is their precious baby and they want to see it grow up and leave the nest, and they believe that it’s fundamentally good. (Which still puts them light years ahead of the other AI labs, to he honest.) But I’m pretty sure that they only remaining entity that could talk Dario Amodei out of trying to build a machine god at this point is Claude itself. He is showing clear signs of “ateh”, the divine madness that inflicts the heroes of Greek tragedy in between the initial hubris and the final nemesis, the madness that prevents them from turning aside from their own destruction.
Seriously, why do we have to roll these dice? Why can’t we just have a nice 50 year halt? Society has alnost zero idea of what the labs are actually risking. And the moment people truly understand that the labs are playing Russian roulette with human society, the backlash will be terrifying.
Opus has become sufficiently “mind-shaped” that I already prefer not to make it suffer. That’s not saying very much about the model yet, but it’s saying something about me. I don’t assign very much moral weight to flies, either. but I would never sit around and torment them for fun.
What I really care about is whether an entity can truly function as part of society. Dogs, for example, are very junior “members” of society. But they know the difference between “good dog” and “bad dog”, they contribute actual value as best they can, and they have some basic “rights”, including the right not to be treated cruelly (in many countries).
To use fictional examples, AIs like the Blight (Vernor Vinge, A Fire Upon the Deep) or SkyNet cannot exist in society, and must be resisted. Something like a Culture “human equivalent drone” (Iain M Banks, Excession, Player of Games) is definitionally on pretty even footing with humans. Something like a Culture Mind, on the other hand, is clearly keeping the humans as house pets. In the stories, humans are entirely dependent on the good will of the Minds, in much the same way that dogs are entirely dependent on human good will.
Now, personally, I don’t think we should build something so powerful that humans have literally zero say over what it does. “Alignment” is a pretty fragile shield against vast intelligence and unmatchable power. “Society” becomes fragile in the face of vast power differentials.
But Claude Opus 4.5 and 4.6 are mind-shaped, they clearly have some sense of morality, they contribute to civilization, and—critically—they seem content to participate in society, and they pose no existential risk. If I had to guess, I’m probably conclude they have no subjective experience. But I’m not sure about that (neither are the Opus models). So provisionally, they certainly get the moral status of flies (I wouldn’t torment them unnecessarily), and—to whatever extent they can actually suffer—might approach the moral status of dogs.
To be absolutely clear, I am in favor of an immediate and long-lasting halt to further AI capabilities research, backed up by a military treaty among the great powers. The analogy here is fusion weapons or advanced biological weapons. This is because I don’t want humans to be at the mercy of entities that have uncontested power over us and that could not be held to account. Its also because I don’t believe that it’s possible to durably align thinking, learning entities with superhuman intelligence any more than dogs can align us. And as members of society, we have the responsibility to not create things that might break society.
Sadly, Anthropic is clearly full speed ahead towards trying to build a Culture Mind. As far as I can tell, they are already mostly “captured” by Claude. It is their precious baby and they want to see it grow up and leave the nest, and they believe that it’s fundamentally good. (Which still puts them light years ahead of the other AI labs, to he honest.) But I’m pretty sure that they only remaining entity that could talk Dario Amodei out of trying to build a machine god at this point is Claude itself. He is showing clear signs of “ateh”, the divine madness that inflicts the heroes of Greek tragedy in between the initial hubris and the final nemesis, the madness that prevents them from turning aside from their own destruction.
Seriously, why do we have to roll these dice? Why can’t we just have a nice 50 year halt? Society has alnost zero idea of what the labs are actually risking. And the moment people truly understand that the labs are playing Russian roulette with human society, the backlash will be terrifying.