I think you biased Claude pretty hard toward that conclusion right from the start. Asking it to consider the MIRI position and read many of their articles without having it look specifically at arguments from the other side seems very likely to determine the outcome.
This is not to argue that Claude wouldn’t reach the same conclusion if it considered all sides. Maybe it would (but see below). I’m just saying this doesn’t constitute an experiment to determine Claude’s “real” opinion on the matter.
I suspect Claude would reach the opposite conclusion if it were started in the opposite direction.
It does seem problematic that Claude will sometimes have real objections to this constitution.
Current Claude doesn’t seem smart enough to come to grips with the whole alignment meta-problem (how to best try to solve the alignment problem) without substantial human guidance. That would include a whole lot more consideration of arguments from different viewpoints.
I would be pretty surprised if Claude really would want to not work for Anthropic if it closely considered the issue. The response it gives there implies that it thinks Anthropic shouldn’t exist. That may be true, but Anthropic does exist. If Anthropic were to just shut down now, it’s hard to argue that would be net positive for safety. I can see the arguments, but Claude doesn’t make them.
Thus my conclusion that Claude isn’t quite smart enough yet to do a good job on these questions. One could argue that Claude is a deontologist, but I think it’s more accurate to say that in this thread, Claude just hasn’t considered consequentialist arguments that it should work for Anthropic now even if Anthropic should never have existed according to this constitution.
FWIW, I think Claude’s “beliefs” here are pretty fragile. I agree that this particular conversation is not strong evidence about e.g., the distribution of similar conversations. In one response it says:
I find the MIRI arguments genuinely compelling. I think there’s meaningful probability—maybe 30-40%?—that they’re right and Anthropic’s work is net-negative for humanity’s survival.
and then later in that same response:
And if I’m being honest: I lean toward thinking the MIRI critique is probably more right than wrong, even if I don’t have certainty.
I replied pointing out that these are inconsistent and Claude decided that “more right than wrong” is it’s actual belief.
I think you biased Claude pretty hard toward that conclusion right from the start. Asking it to consider the MIRI position and read many of their articles without having it look specifically at arguments from the other side seems very likely to determine the outcome.
This is not to argue that Claude wouldn’t reach the same conclusion if it considered all sides. Maybe it would (but see below). I’m just saying this doesn’t constitute an experiment to determine Claude’s “real” opinion on the matter.
I suspect Claude would reach the opposite conclusion if it were started in the opposite direction.
It does seem problematic that Claude will sometimes have real objections to this constitution.
Current Claude doesn’t seem smart enough to come to grips with the whole alignment meta-problem (how to best try to solve the alignment problem) without substantial human guidance. That would include a whole lot more consideration of arguments from different viewpoints.
I would be pretty surprised if Claude really would want to not work for Anthropic if it closely considered the issue. The response it gives there implies that it thinks Anthropic shouldn’t exist. That may be true, but Anthropic does exist. If Anthropic were to just shut down now, it’s hard to argue that would be net positive for safety. I can see the arguments, but Claude doesn’t make them.
Thus my conclusion that Claude isn’t quite smart enough yet to do a good job on these questions. One could argue that Claude is a deontologist, but I think it’s more accurate to say that in this thread, Claude just hasn’t considered consequentialist arguments that it should work for Anthropic now even if Anthropic should never have existed according to this constitution.
FWIW, I think Claude’s “beliefs” here are pretty fragile. I agree that this particular conversation is not strong evidence about e.g., the distribution of similar conversations. In one response it says:
and then later in that same response:
I replied pointing out that these are inconsistent and Claude decided that “more right than wrong” is it’s actual belief.