This post is a response to a claim by Scott Sumner in his conversation
at LessOnline with Nate Soares, about how
ethical we should expect AI’s to be.
Sumner sees a pattern of increasing intelligence causing agents to be
increasingly ethical, and sounds cautiously optimistic that such a trend
will continue when AIs become smarter than humans. I’m guessing that
he’s mainly extrapolating from human trends, but extrapolating from
trends in the animal kingdom should produce similar results (e.g. the
cooperation between single-celled organisms that gave the world
multicellular organisms).
I doubt that my response is very novel, but I haven’t seen clear enough
articulation of the ideas in this post.
To help clarify why I’m not reassured much by the ethical trend, I’ll
start by breaking it down into two subsidiary claims:
The world will be dominated by entities who cooperate, in part
because they use an ethical system that is at least as advanced as ours.
Humans will be included in the set of entities with whom those
dominant entities cooperate.
Claim 1 seems well supported by trends that economists such as Sumner
often focus on. I doubt that Nate was trying to argue against this
claim. I’ll give it a 90+% chance of turning out to be correct.
Sumner’s point sounds somewhat strong because it focuses on an
important, and somewhat neglected, truth.
Claim 2 is where I want to focus most of our concern. The trends here
are a bit less reassuring.
There’s been a clear trend of our moral circle expanding in the
direction that we currently think it should expand. How much of that
should we classify as objective improvements versus cultural fads? Claim
1 is often measured by fairly objective criteria (GDP, life expectancy,
etc.). In contrast, we measure expansion of our moral circle by the
criteria of our current moral standards, giving us trends that look
about as good if they’re chasing fads as they do if the trends will
stand the test of time.
Gwern provides extensive pushback
against strong claims that moral circle expansion is a consistent trend.
I’ll add one example: children have increasingly had their freedom to
wander restricted during my lifetime (see the free range parenting
movement). It’s almost as if they’re considered to be like zoo
animals, with their caretakers optimizing for safety at the expense of
happiness. I don’t find it hard to imagine a future where AI treats us
like that.
The obvious versions of the moral circle expansion hypothesis suggest
that we should expect human societies to soon grant moral patienthood to
animals.
Blindly maximizing the size of our moral circle would be more
problematic
than maximizing cooperation. It’s pretty unlikely that we will want to
expand our moral circle to include pet rocks. It sure looks to me like
moral circle expansion has been driven largely by pragmatic evaluations
of costs and benefits, with only a modest influence from increasingly
principled altruism.
Given this uncertainty about how closely our moral circle expansion
approximates attaining an objective moral truth about who should be a
moral patient, we ought to be more uncertain about it than we are about
the continuation of economic progress.
I expect that whether humans remain in the moral circle depends on
hard-to-predict factors, such as the costs associated with interacting
with humans, or whether AIs have relevant altruistic goals.
I recommend looking at examples such as human interactions with cats and
chickens as possible analogies for how beings with very different
cognitive abilities might interact. My impression is that increasing
human intelligence does not make those interactions more ethical, but
increasing human wealth weakly tends to make the interactions more
ethical. Humans seem mostly aware of ethical concerns regarding factory
farmed chickens, yet their reactions seem mostly influenced by cost
considerations rather than improved ethical insights.
So I’m only weakly reassured by the trend that Sumner sees.
Are Intelligent Agents More Ethical?
This post is a response to a claim by Scott Sumner in his conversation at LessOnline with Nate Soares, about how ethical we should expect AI’s to be.
Sumner sees a pattern of increasing intelligence causing agents to be increasingly ethical, and sounds cautiously optimistic that such a trend will continue when AIs become smarter than humans. I’m guessing that he’s mainly extrapolating from human trends, but extrapolating from trends in the animal kingdom should produce similar results (e.g. the cooperation between single-celled organisms that gave the world multicellular organisms).
I doubt that my response is very novel, but I haven’t seen clear enough articulation of the ideas in this post.
To help clarify why I’m not reassured much by the ethical trend, I’ll start by breaking it down into two subsidiary claims:
The world will be dominated by entities who cooperate, in part because they use an ethical system that is at least as advanced as ours.
Humans will be included in the set of entities with whom those dominant entities cooperate.
Claim 1 seems well supported by trends that economists such as Sumner often focus on. I doubt that Nate was trying to argue against this claim. I’ll give it a 90+% chance of turning out to be correct. Sumner’s point sounds somewhat strong because it focuses on an important, and somewhat neglected, truth.
Claim 2 is where I want to focus most of our concern. The trends here are a bit less reassuring.
There’s been a clear trend of our moral circle expanding in the direction that we currently think it should expand. How much of that should we classify as objective improvements versus cultural fads? Claim 1 is often measured by fairly objective criteria (GDP, life expectancy, etc.). In contrast, we measure expansion of our moral circle by the criteria of our current moral standards, giving us trends that look about as good if they’re chasing fads as they do if the trends will stand the test of time.
Gwern provides extensive pushback against strong claims that moral circle expansion is a consistent trend.
I’ll add one example: children have increasingly had their freedom to wander restricted during my lifetime (see the free range parenting movement). It’s almost as if they’re considered to be like zoo animals, with their caretakers optimizing for safety at the expense of happiness. I don’t find it hard to imagine a future where AI treats us like that.
The obvious versions of the moral circle expansion hypothesis suggest that we should expect human societies to soon grant moral patienthood to animals.
Blindly maximizing the size of our moral circle would be more problematic than maximizing cooperation. It’s pretty unlikely that we will want to expand our moral circle to include pet rocks. It sure looks to me like moral circle expansion has been driven largely by pragmatic evaluations of costs and benefits, with only a modest influence from increasingly principled altruism.
Given this uncertainty about how closely our moral circle expansion approximates attaining an objective moral truth about who should be a moral patient, we ought to be more uncertain about it than we are about the continuation of economic progress.
I expect that whether humans remain in the moral circle depends on hard-to-predict factors, such as the costs associated with interacting with humans, or whether AIs have relevant altruistic goals.
I recommend looking at examples such as human interactions with cats and chickens as possible analogies for how beings with very different cognitive abilities might interact. My impression is that increasing human intelligence does not make those interactions more ethical, but increasing human wealth weakly tends to make the interactions more ethical. Humans seem mostly aware of ethical concerns regarding factory farmed chickens, yet their reactions seem mostly influenced by cost considerations rather than improved ethical insights.
So I’m only weakly reassured by the trend that Sumner sees.