Notes on the possibility of moral progress
To achieve the best possible future, we must know what that future looks like. In other words, we need to solve ethics.[1]
The problem of solving ethics is so large and abstract that it’s difficult to say useful things about. In lieu of any structured analysis, herein lies a collection of thoughts about the problem.
Cross-posted from my website.
Acting in the face of moral uncertainty
If we have persistent moral uncertainty between maximizing and satisficing[2] moral theories, then it’s not difficult to decide what to do in practice: allocate a tiny portion of the universe to satisfying the non-maximizing moral theories, and allocate the rest to the maximizing moral theories.
Example: If theory A says we should fill the universe with welfareans, and theory B says we should preserve homo sapiens but it doesn’t much matter how many humans there are, then we can near-perfectly satisfy both theories by maintaining humanity in a small segment of the universe and giving the rest to welfareans.
However, the distinction between maximizing and satisficing theories may be irrelevant because plausible satisficing theories still hold that it is a good thing to maximize The Good, even though it is not morally obligatory[3] to do so. Satisficing theories would still want to fill most of the universe with The Good.
Given uncertainty between mutually incompatible maximizing moral theories, we have to choose. Allocating resources incorrectly would be catastrophically bad.
Can we discover facts that resolve moral uncertainty?
I believe so. I expect that we can eventually eliminate almost all moral disagreements purely by discovering facts. The fact-value distinction implies that we cannot 100% determine what we ought to do by discovering facts, but most of what look like terminal values disagreements are not truly terminal.[4]
Some examples:
Right now, we do not know how to weight different people’s experiences against each other. I strongly suspect that there are facts of the matter about how to weight experiences, and that these facts can be discovered empirically.
The problem of weighting experiences is downstream of the hard problem of consciousness. I likewise suspect that there is a definitive answer to the hard problem,[5] and that we can, in principle, find that answer.
What is the nature of personal identity? Certain theories of personal identity rule out classes of moral theories. If personal identity is not metaphysically meaningful, then person-affecting views must be false—there is no relevant distinction between bringing a new person into existence and changing the life trajectory of an already-existing person. Changing a life trajectory creates new person-moments, which is (in this view) metaphysically equivalent to creating a new person.
There may be some way to salvage person-affecting views, but if so, that possibility would itself be a non-normative fact—i.e., person-affecting views may be permitted or ruled out purely by facts, without any moral stance required.
Another important (although slightly obscure) question is: given two identical copies of the same mind, are they experiencing “twice as much” as if there were only one copy, or “the same amount”?[6] This seems like a factual question, not a moral one. I have no idea how we would answer this question, but it seems answerable in principle.
Harsanyi’s utilitarian theorem (see also Harsanyi (1955)[7]), which showed that if individuals have VNM utility functions, and if the Pareto principle[8] holds over groups, then a version of utilitarianism must be true. The Pareto principle is a normative principle, not a factual one; the question of whether individuals ought to care about their own welfare is also a normative one. But I strongly suspect that there is a fact of the matter about whether an individual’s welfare can be described as a utility function, and there is a fact of the matter about how exactly that function is specified. Discovering those facts would get us at least part of the way toward a fully-specified theory of ethics.
It matters whether the universe is finite or infinite, and whether our actions can have finite or infinite influence. Infinite ethics poses troubling problems, but some (maybe all) of those problems can be resolved factually (or if they’re unsolvable, then their unsolvability is a factual question).
Note: I don’t think they can be resolved purely empirically. For example, by our understanding of the laws of physics, our actions cannot have infinite influence.[9] But there is a nonzero probability that we are wrong about the laws of physics and that our actions can have infinite influence after all. No amount of empirical investigation can reduce our uncertainty to zero, but there may nonetheless be a mathematical solution to the problem.
Standard formulations of deontology may break down in light of the fact that you do not have certainty about the consequences of your actions, so you can never be sure that you’re not doing something impermissible by acting. (See also Nye (2014)[10].) If so, those flavors of deontology are ruled out purely based on a factual analysis (no normative claims necessary).
Many value disagreements do not purely boil down to a disagreement about facts, but I expect they can be resolved anyway. Some examples:
People would rather donate money to a single identifiable person than to a much larger, but nebulous, group of people. This preference should break down upon reflection. Suppose Alice is offered the chance to donate to a single identifiable person. Then imagine an alternative world where Alice can donate the same amount of money to help that same single person plus several other people, but the single person is never identified. Surely she would prefer this.
I believe the disagreement between negative utilitarians and classical utilitarians[11] would be resolved if we knew how to directly compare experiences / if we solved the hard problem of consciousness.
My guess is that negative utilitarianism is a mistake stemming from the fact that maximum suffering in humans far exceeds maximum happiness, and this creates the appearance that suffering is terminally more important than happiness.
People support animal welfare, but also eat factory-farmed animals. It’s conceivable that people in reflective equilibrium would resolve this inconsistency by throwing out their concern for animal welfare, but that seems unlikely.
I believe people reject the mere addition paradox due to scope insensitivity—an enormous population of slightly happy people is indeed better than a small population of very happy people. It should be possible to prove that this intuition is the result of scope insensitivity, that scope insensitivity is inconsistent with people’s other values.
Alternatively, some argue that an enormous population of slightly happy people is not particularly good because the experiences are too uniform, and two copies of an identical experience is no better than one copy. If there is a fact of the matter about how to consider two copies of an experience, then this alternative view could be proven right.
Our understanding of philosophy is limited. What does it mean to do good philosophy? What qualifies as a good philosophical argument?
We could make progress on those questions. We have already made progress: Descartes innovated on rightly conducting reasoning and seeking truth. The significance of philosophical thought experiments is a recent development—the concept is pre-Socratic, but modern thought experiments are more refined and more useful. (On Wikipedia’s list of notable philosophy thought experiments, two thirds were invented after 1900, and over half were not developed until 1960 or later.[12]) Most modern concepts in moral philosophy come from the 1700s or later; philosophy of mind primarily comes from the 1900s;[13] analytic philosophy improved on the methods of its predecessors, and did not emerge until the 1800s. All that suggests that civilization is indeed making progress on philosophy, even if the rate of progress is slow.
Some normative claims evade fact-based analysis
I hold some foundational moral beliefs that seem unrelated to descriptive facts. I cannot conceive of how an empirical or theoretical investigation could provide reasons to believe that these are true or false.
It seems self-evident that pleasurable experiences are good (and suffering is bad), in the same way it’s self-evident that I am conscious. This is difficult to dispute, and to my knowledge virtually all moral philosophers (and regular people) agree that pleasure is good and suffering is bad.
I find it hard to see how anything other than good or bad experiences could be good or bad, because where does the goodness or badness come from if it’s not being directly experienced by anyone?[14] (A.K.A. welfarism.) However, many people believe that non-experiences can be innately good or bad, and I don’t see how we could resolve this dispute.
Other people’s experiences matter, not just my own. (This claim is uncontroversial, but still, I see no way to prove it, or even give any reason to believe that it’s true.)
All beings’ experiences matter equally. It does not matter who the experience resides in; all that matters is the intensity of the experience. This view has several corollaries:
Welfare aggregates linearly across individuals.
The total view of population ethics is correct.
Speciesism is wrong—experiences of humans should not be given more moral weight purely due to species membership.
If you live in the 1700s, it implies that slavery and misogyny are wrong.
The first and third claims (the goodness of pleasure/badness of suffering, and the principle of altruism) are widely accepted. Many people disagree with me about the second and fourth claims, and there is no visible path to resolving those disagreements—plus, I have internal uncertainty about whether they’re true, which I have no idea how to resolve.
Even though the first and third claims are uncontroversial, they still evade any attempt to explain why they’re true. It could be that we’re all wrong.
And yet, it’s possible to convince people about these sorts of normative claims. (Peter Singer made arguments that convinced many people, including me, of the principle of equal consideration of interests.) What’s going on inside people’s heads when they change their minds about seemingly terminal values? Or, what’s going on when I contemplate two conflicting intuitions and decide that one is more important than the other? We have no theory of what constitutes a good argument for a normative position. We have some idea about the sorts of arguments people find convincing, but not a great understanding of why, and no way of saying that people are right to be convinced by a particular argument.
Implications for how the future goes
This essay has posited that we are making progress in philosophy, and that most (maybe all) moral disagreements can be resolved by learning new facts. If true, what does that imply?
Smarter-than-human AI should be better than humans at discovering facts. That’s useful insofar as moral disagreements can be resolved by facts.
The positive vision in the epilogue of AI 2040 has every individual human controlling an equal part of the lightcone. How good an outcome is that? If it’s feasible to converge on moral beliefs, then in that scenario, people (with superintelligent AI assistants) will come to agree on ethics, and will shape the universe in the way it ought to be shaped. Some people may have persistently bad values (like maybe Putin, or maybe not), but if most people converge on good values, then most of the universe will be directed well.
How well would a Long Reflection work? It would provide more opportunity to discover ethics-relevant facts and to improve our understanding of how to do good philosophy. But it would also give amoral actors more time to seize power. The AI 2040 idea of “give everyone an equal share of the lightcone, and then let people cooperate if they want to” is plausibly better than a Long Reflection, and plausibly worse.
Over sufficiently long time horizons, natural selection takes over. The dominant ethical belief will be that the right thing to do is to spread one’s own genes at the exclusion of everything else. We need to solve ethics before that happens, or otherwise prevent that from happening.[15]
In the scenario where humans control the future, the principle I worry about most is impartial altruism. I worry that most of the people in control will simply not care to devote resources to helping others.
I worry much less about disagreements between altruistic people (e.g., between negative and classical utilitarians). I expect these disagreements can be resolved factually.
Notes
- ^
Or perhaps it would be more accurate to speak of solving axiology, i.e., “what is good?” as opposed to “what is right?”
- ^
A maximizing theory holds that the right things to do is to maximize some quantity—usually, to maximize utility, although “utility” can be defined in various ways. A satisficing theory says that there are certain things we ought to do (e.g. don’t commit murder), but as long as we do those, we have “satisfied” our moral obligations.
- ^
I’m hesitant to use the word “obligatory” because it creates confusion when talking about consequentialist theories. For more on this, see Richard Y. Chappell’s blog post Deontic Pluralism (2022) or his academic paper Deontic Pluralism and the Right Amount of Good (2020).
- ^
There is a class of moral disagreements that can clearly be resolved by factual questions. For example, should I drive a bulldozer through a particular building? That entirely depends on the answer to the factual of “is this an empty run-down building that’s scheduled to be demolished, or it a house that somebody’s living in?” Those sorts of disagreements are not interesting for the purposes of this essay.
- ^
Some people claim that there is no fact of the matter about the hard problem of consciousness. I find it hard to comprehend why anyone holds that view. There is, at minimum, a fact of the matter about whether I am conscious (and that fact is “I am conscious”). I find this position about as confusing as the illusionist view (i.e. the view that consciousness is an illusion and in fact there is no such thing as consciousness).
- ^
Bostrom, N. (2006). Quantity of experience: brain-duplication and degrees of consciousness.
- ^
Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility.
- ^
The Pareto principle states that if outcome A is at least as good as outcome B for every person, and outcome A is better for at least one person, then outcome A is better overall.
- ^
Sandberg, A. & Manheim, D. (2021) What Is the Upper Limit of Value?
- ^
Nye, H. (2014). Chaos and Constraints.
- ^
Here, classical utilitaranism refers to any flavor of utilitarianism that gives equal weight to happiness and suffering.
- ^
Dates are pulled from Wikipedia and aggregated by Claude Opus 4.8 (chat source). Of the 41 philosophy thought experiments on Wikipedia’s list, there are 13 from pre-1900, 5 from 1900–1959, 20 from 1960–1989, and 3 from 1990–present. I’m taking Wikipedia’s list as a reasonable proxy for notability.
- ^
Thomas Nagel’s What Is It Like to Be a Bat?, written in 1974, is a more lucid exploration of consciousness than anything that came before it, and represented important progress. More broadly, much of the best work on philosophy of mind came from people who are still alive today.
- ^
The parable of the Hrogmorph’s Heartstone is an attempt at justifying this intuition. (See also the extended parable.)
- ^
Natural selection hasn’t taken over yet because humans are adaptation-executors, not fitness-maximizers. We have evolved the intelligence necessary to explicitly optimize for genetic fitness, but our big brains haven’t been around long enough for natural selection to push us in that direction.
Why do you think we can find an answer to the hard problem of consciousness? You argue (to me convincingly) that it’s bizarre to claim there is no fact of the matter. But I think the conventional view is that there is a fact of the matter but finding it will be very, umm, hard.
(I am fond of the “meta hard problem of consciousness”, namely why you would say you think there’s a hard problem of consciousness. In principle that should be solvable..)
I’m not confident we can find an answer. But we’ve made progress in the last 50–100 years:
Dualism was (approximately) shown to be false through pure a priori reasoning, but nobody figured it out until the 1900s.
Neuroscientific evidence is widely regarded as evidence against dualism (although IMO the a priori arguments are more important).
Computational functionalism is IMO the strongest theory, but it wasn’t even possible to formulate this theory until we had a theory of computing, which was developed in the early-mid 1900s.
I gave a couple other relevant examples in OP.
There’s also the “working backwards from beliefs to evidence” argument: I’m confident that Dynomight is conscious; therefore, I must have evidence that he’s conscious. I’m less sure about what that evidence is, but the problem of “figure out what evidence I used to determine that Dynomight is conscious” seems like a solvable (if difficult) problem. (I can point to a bunch of pieces of evidence, but I don’t know how to weight them, or if there’s more evidence I haven’t pointed to.)
From a pure outside view, I guess I buy it. It seems hard, but we’ve made progress on lots of other stuff that seemed hard.
But I think the inside view is very problematic here! You have zero access to any consciousness except your own. It seems like this makes it ~impossible to learn anything through experimentation. For example, don’t we already know why you think I’m conscious? It’s because (A) you know you’re conscious (very strong evidence) plus I seem similar to you and (B) some assumption that your place in the universe isn’t that unique (no real evidence, but plausible).
The list provided shows a lot of future empirical questions whose answers will influence ethical theories, but there may also be some interesting past examples already. For example, the Theory of Evolution showing how humans and animals are not different in kind likely played a supporting role in animal welfare. Going back even further, many ancient cultures believed that certain diseases were a sign of either divine punishment or something otherwise supernatural, which we now know is not true thanks to our modern medical science.
The linked article on Long Reflection suggests rather large timescales (1000+ years in certain cases), but since you had mentioned what seems like an accelerating rate of philosophical progress (e.g. the fact that >1/2 of the thought experiments on Wikipedia’s list came after 1960), I’m not entirely sure that humanity would actually have to spend that long before either solving ethics, or at least generally agreeing on immediate next steps. If we presume that at least part of this acceleration exists independently of our recent technological advancement—assuming that a Long Reflection involves a pause in at least some respects, namely advanced AI—then it might not be that long for another doubling in thought experiments / general philosophical output to occur. In summary, even if humanity embarks on a Long Reflection with the intention of it lasting a very long time, I find it plausible that 20-50 years down the line, they may have already accumulated enough knowledge to at least make some changes to the world, and then debate next steps from a new starting point. During a reflection of that length, it is still very much possible for amoral actors to gain more power, but I would consider it less of a risk than with the original timescales.
Overall, I generally agree with your implications for the future, and share your concern about whether those in power will care about others, which seems like a more challenging issue to address as it is difficult to imagine ways to improve our chances on an issue of personal motivation.
Yeah I find it pretty plausible that without AGI, humanity would “solve ethics” within the next 50–100 years. Although there may also be a wide gap between philosophers converging on a solution and society “catching up” to philosophers.
(Today, philosophers overwhelmingly reject dualism, and the arguments against dualism have pretty much defeated it, but I believe a majority of people are still dualists.)
I notice you seem to be assuming something like utilitarian metaethics—you don’t see any kind of open questions about deontology or virtue ethics. That seems to have a bearing on your bold claim that that value can be reduced to fact.
The next sentence takes about discovering how much agents want different things, …
...as if the way the reduction will work is a weighted compromise. But that would depend on the correct metaethics. According to many versions of deontology, some things are just plumb wrong.snd you shouldn’t do them at all no matter how many people like them.
I’m not sure what you mean by “utilitarian metaethics” but I’d say my post is mainly about axiology, and any reasonable moral theory has a consequences-oriented axiology. Deontology and virtue ethics theories agree with consequentialism that it’s good for there to be more good stuff. There can be different theories of what “good stuff” means, but a particular deontologist and a particular consequentialist might be in 100% agreement about that question.
On the question of weighting experiences, I hadn’t thought about this much til you brought it up but here’s my current perspective:
Take chicken welfare as an example. Some theory of deontology might say that it’s wrong to factory-farm chickens if they experience any suffering, regardless of the degree of that suffering; whereas a utilitarian might say chicken farming is okay if their suffering is sufficiently low-level. So both care about chicken experiences, and therefore the fact of the matter about chicken experiences is relevant, but for that deontological theory it’s a binary question, not a weighting.
Alternatively, a deontological theory might say that causing suffering is permissible if you get a sufficiently large benefit in exchange (as I understand, this view is popular among deontologist philosophers). That theory does want to know the magnitude of chicken suffering, because factory farming might be permissible if the level of suffering is low enough that it can be outweighted by the benefits to humans. (FWIW I find it pretty implausible that it would be, but I’m sure I could come up with some other example scenario where it’s more plausible.) My point being that deontologists probably also care about the (probably-)factual question of the magnitudes of experiences of different beings.
I don’t agree with this, because of the blissful ignorance problem.
I have preferences about the state of the world. If I am fooled into believing that the state of the world matches those preferences, I experience as much pleasure as I do if it actually matches those preferences. Yet I would say that being fooled is bad, and certainly that it at least is worse than having those preferences actually be satisfied.
My main issue is that the ancestral environment or a historical environment had religions declare a similar goal and fail to favor the beliefs which you mocked. For example, The Bible had God say: “Be fruitful and multiply, and fill the earth and subdue it; rule over the fish of the sea and the birds of the air and every creature that crawls upon the earth.” In the ancestral/historical environment producing kids was bottlenecked on the ability to satisfy needs like food or housing, which in turn was bottlenecked on the community’s ability to guard resources, to transform them into useful outcomes and to avoid overexploiting the resources.