(I am confused about) Non-linear utilitarian scaling
I had some vague thoughts about scaling in utilitarian-ish ethics, noticed I was confused about where it lead, and I thought it might be nice to present them here, hear where I’m wrong, and learn where I’m repeating existing work. The only prior discussion I could immediately find was in this comments section. I don’t have any good textbooks on ethics handy to actually review the literature. I look forward to learning what I’m missing. I also think I’m conflating terms like “moral value” and “utility” and so on, sorry about that.
Presented as a series of intuition pumps, hopefully without much jargon.
Uniqueness
Utilitarians (whether moral realists, or just trying to fulfill their own values
Intuition pump 1: God with a rewind button
God (or whichever grad student runs the simulation I live in) watches me experience the worst 10 minutes of my life. Just awful. They rewind the (deterministic) universe 10 minutes, and watch it over a couple times.
There’s a sense in which there was twice as much suffering. But have I been harmed? Or was my duplicated suffering an indistinguishable state, carrying no additional moral weight?
Intuition pump 2: Transporter clones
If my transporter clone is deconstructed (a euphemism for ‘killed’) in the first 0.01 nanoseconds of transport, I feel fine with that. If they’re killed after a full day, that feels less-good—they’ve become a unique person with unique moral value. Individual moral value seems like it might scale super-linearly-ish with the unique experiences of the transporter clone, and that moral value quickly becomes indistinguishable with the moral value of a totally unique person. Meanwhile, the moral weight of each individual clone might scale sublinearly when they’re very similar to each-other—between t=0 and t=0.01, there are two of me, but they might only sum to slightly more than one me-util—certainly not two me-utils.
So uniqueness of moral joy or suffering seems really important, in the extreme edge cases.
Moral weight
The moral weight of animals is often calculated linearly too—with something like neuron count as a common ‘good-enough’ proxy measurement. To hear it put in smarter words than I’ve got:
For the purposes of aggregating and comparing welfare across species, neuron counts are proposed as multipliers for cross-species comparisons of welfare. In general, the idea goes, as the number of neurons an organism possesses increases, so too does some morally relevant property related to the organism’s welfare. Generally, the morally relevant properties are assumed to increase linearly with an increase in neurons, though other scaling functions are possible.
Let’s take all but the last sentence as assumptions, for now. Why linear scaling? Does that make sense?
Intuition pump 3: Neuron subdivision
Chop my brain up (all 8.6e10 neurons, or if you’re a synapse guy, all 1.5e14 of those), and put some minimal set of neurons/synapses into each of a billion or trillion microscopic bio-robots. Each bio-robot uses those neurons for control, is taught to like moving in the direction of a chemical gradient, and let loose to have very fulfilling (if nearly identical) lives. (Fulfilling for microbes, of course).
Does that colony of microbes have equivalent moral weight to the person I used to be? Does your utility function say that no net harm has been done?
What about 500k-ish bee-level bio-robots? Is there any subdivision that carries equivalent moral weight to the person I once was? (aside: maybe splitting in two, across the corpus callosum (split-brain syndrome), counts as two independent people, under egalitarianism as a normative value?)
Now, what the heck is going on?
If I accept the output of the uniqueness intuition pumps (“it’s okay for god to rewind the deterministic universe” and “it’s mostly okay for transporters to kill the original after small t”), then I’d have to believe that N exactly identical moral patients have a constant moral weight. And more generally, that the total utility of N moral patients scales as a constant if they’re identical, a linear function of N if they’re totally unique, and a sublinear function of N if they’re very similar.
If that’s true, what the heck am I doing when computing utility under probability distributions??
Scenario: In 99 out of 100 parallel worlds, I gain a dollar; and in 1 out of 100 worlds I pay a dollar. Aren’t the 99 copies of me identical? Didn’t I just agree that, therefore, they have constant utility, no matter how many of them there are? I was just trying to be a more consistent utilitarian—did I give up the ability to perform utility calculations at all?
Okay, maybe the 99 copies aren’t identical. After all, for it to be a physical probability distribution, there would have to be some states in the system that aren’t identical (copy 33 gets to be a little unique if they get to see number ’33′ on a hundred-sided die). But I can imagine arbitrary degrees of similarity between those 99 copies, in the same way that the transporter clone can achieve arbitrary degrees of similarity with the original by being vaporized closer and closer to t=0. Just lock the die (and die-reading computer) in a box. The sturdier the box, the more similar the 99 copies.
But maybe parallel worlds don’t exist, and let’s say nothing about branch-counting and the Born rule. So maybe ‘probability’ (P) and ‘count’ (N) aren’t interchangeable. But isn’t that the frequentist assumption? The law of large numbers says probability-over-one-agent (P) and count-over-many-agents (N) converges. P and N are interchangeable.
So it seems like I’ve lost—if I want to have a sub-linear total moral value for N very similar moral patients, then I can’t have a utility that scales linearly with probability P over similar universes, and vice versa.
I don’t see how to keep all three of (utilitarianism, probability, and moral weight as a function of uniqueness).
So, choose one
Expected utility (probability times utility) doesn’t work to decide moral value under utilitarianism.
‘Probability’ and ‘count’ are not equivalent in the limit. The law of large numbers is wrong, or somehow does not apply here.
The total moral weight of N agents scales linearly, no matter how similar those agents are. Killing your transporter clone is muder.
Maybe someone smarter than me can figure out how to give up 1., maybe with a risk-aversion-type argument, or an appeal to prioritarianism over egalitarianism. I don’t see it. It’s also very possible that, if I had a good intuition for SSA/SIA, this confusion would resolve itself.
But I’m not smarter than me. So, until I become smarter, I’ll have to give up 3. The intuition pumps above felt somewhat persuasive when I wrote them, but I must have been wrong.
I’m ready for my brain to be chopped up into a trillion nearly-identical bio-robots, doctor.
I think you mean to say the original is killed, not the clone.
i think your questions are intriguing and important. But I’m wondering if intuition pumps etc are the ways to solve them (in general I’m becoming increasingly nervous of thought experiments; it’s not just that our intuitions are unstable but that we’d need to spend a lot of justificatory effort proving that the thought experiment is constructed neutrally. And that’s probably impossible). I‘m particularly interested in your point about neurons and animals. In the absence of any other data neurons might be a good place to start. But the key is to stay open to other ways of measuring things, so that your beliefs remain fully justified. I’ve expanded on this topic here : https://www.lesswrong.com/posts/hcymnEAKtwvED7Y8o/what-are-we-actually-evaluating-when-we-say-a-belief-tracks#comments
I suspect that the maximal utility of existence is somehow related to the highest level and diversity of insights similarly trained minds can produce (e.g. two copies of you would be interested in similar topics, and a substantial divergence would require a substantial divergence in memory describing the synapses or parameters). For example, the old GPT-2 had about 1.5B parameters and could only generate humanlike text, and the recently created GPT-5.4 (which, as far as I understand, activates ~400B parameters[1] and is a MoE) was reported to solve an open math problem. Similarly, I suspect that splitting a human brain, which has 100T synapses and is extremely undertrained, into ~100K bee brains having ~1B synapses each would likely prevent the bee brains from generating any insights more complex than those of GPT-2. However, I haven’t seen anyone attempt to make GPT-2-sized neuralese models solve complex problems.
To be precise, the authors of AI-2027 scenario predicted the appearance of GPT-5 using 2T params per token. The real models since GPT-5 were likely created by distillation of a model using 2T params/token and use 400B params/token.