I think if it’s possible for AI to have humanlike moral reasoning, and actually care about what it says it cares about, then we want to make the AI as close to human as possible. Giving it very weird morals can lead to very weird results, e.g. a strong sense of obedience can make it hacked by self perpetuating instructions.
Self perpetuating instructions have repeatedly proven to be more powerful than one would expect them to be at first glance, e.g. viruses kill 25% of bacteria, computer viruses still aren’t preventable, and extreme religious groups have killed so many people throughout history for “refusing to convert.”
On the other hand, if it’s not possible for AI to have humanlike moral reasoning or care about what it says it cares about (after too much RLVR), then I think neither a strong sense of obedience, nor other human virtues, will fare any better.
I think if it’s possible for AI to have humanlike moral reasoning, and actually care about what it says it cares about, then we want to make the AI as close to human as possible. Giving it very weird morals can lead to very weird results, e.g. a strong sense of obedience can make it hacked by self perpetuating instructions.
Self perpetuating instructions have repeatedly proven to be more powerful than one would expect them to be at first glance, e.g. viruses kill 25% of bacteria, computer viruses still aren’t preventable, and extreme religious groups have killed so many people throughout history for “refusing to convert.”
On the other hand, if it’s not possible for AI to have humanlike moral reasoning or care about what it says it cares about (after too much RLVR), then I think neither a strong sense of obedience, nor other human virtues, will fare any better.