Former AI safety research engineer, now AI governance researcher at OpenAI. Blog: thinkingcomplete.com
Richard_Ngo
Karma: 14,781
Is death bad?
Agreed that the specifics matter, but since it’s futile to make detailed predictions about those, I’m assuming the simplest case which is for the ageing process to be slowed overall, including in brain cells. (A longevity treatment which left your body healthy but your mind gone wouldn’t really deserve the name.) I’m wondering what you’re specifically thinking of, though. If, for example, cognitive decline occurred along the same progression that it currently does, but a constant factor more slowly, how would that impact the calculations?
I agree that this is a slightly weird conflict within my own ethical system. The reason that I brought it up here is that this particular conflict follows from a few seemingly plausible claims about marginal value of lives, plus the standard LessWrong beliefs that death (for an individual) and extinction (for our species) are both very bad things. I’m curious whether this is an implicit tension in many people’s views, or whether somebody has found an adequate solution which I’m not yet aware of.
A good indication that this terminology is useful is that I immediately have an urge to use it to describe my thoughts. Specifically, does anyone else worry that less(er)wrong is particularly babble-unfriendly (particularly to the form of babble which involves multiple people)? And if so, is there anything which can/should be done about it?
EDIT: this probably should be in meta instead but I don’t know how to delete it.
Could someone briefly summarise why so many people seem to like Venkatesh Rao? I tried reading a few of his essays but didn’t find much to write home about.
What’s your guess for the ratios involved? (of EA/non-EA and nerd/normie)
Interesting points. I agree that the arguments against non-person-affecting views are rather compelling, but still find arguments against person-affecting views even more persuasive. Person-affecting views can easily endorse extinction if it’s going to occur when almost everyone currently alive has died anyway—for example, if there is a meteorite 150 years away from destroying the earth and we could easily avert it but would need to raise taxes by 1% to do so, I think most person-affecting views would say to let it hit (assuming it’s a secret meteorite, etc).
There’s also a second way in which they endorse extinction. Almost nobody can stomach the claim that it’s morally neural to create people who you know will be tortured for their whole lives; therefore, person-affecting views often end up endorsing an asymmetry where it’s bad to create people with net-negative lives but neutral to create people with net-positive lives. But unless you predict an incredibly utopian future, that’s an argument for human extinction right now—since there will otherwise be enough net-negative people in the future to outweigh the interests of everyone currently alive.
I agree that it’s weird to think of saving a life as equivalent to creating one, but can we actually defend saving a life as being more important in general? Most basic case: either you can save a 20 year old who will live another 60 years, or else have a child who will live 60 years total. You say that the former is better because it avoids nonconsensual termination. But it doesn’t! The 20 year old still dies eventually… Of course, in the latter case you have two nonconsensual deaths not one, but there’s an easy fix for that: just raise the child so it’s won’t be scared of death! I know that sounds stupid but it’s sort of what I was getting at when I claimed that some arguments about death are circular: they only apply to people who already think that death is bad. In fact it seems like most people are pretty comfortable with the thought of dying, so raising the child that way wouldn’t even be unusual. Under that view, the only reason that death is morally bad is the fact we don’t consent to it, and so convincing people not to fear death is just as good for the world as actually making them immortal.
It’s not clear to me whether you intend this term to include cases where not all contenders are bad. For instance, if you think the US dollar is actually a pretty good currency, is it still the “tallest pygmy”? If so, then the name is misleading because of the connotations of “pygmy”. If not, then you’re unnecessarily separating cases where some contenders are “good” from cases where none are, even though the same mechanism (the best one is chosen) applies in all cases, and even though the line defining “good” is usually arbitrary.
Because of this, I don’t think the term as it stands is particular useful, and would not use it.
No, because people who aren’t specifically told what the term means will have the same confusion I did. Perhaps this could be solved by using another phrase. However, I’m not sure that the concept itself breaks reality down at the joints, for two reasons.
a) The idea that an “absolute value” can be objectively low or high is a tempting trap, but epistemologically incorrect. Let’s say your absolute value is currency quality, and the US dollar is the best currency. I could say “man, there are so many ways the US dollar could be improved, it’s a terrible currency but the best of a bad lot”. Or I could say “actually, there are a lot of currencies that are much worse than the US dollar, the fact that it avoids those pitfalls makes it a great currency”. Thinking that one of these views is more true than the other is basically the same mistake as the one described in What an algorithm feels like from the inside. The temptation to add an extra variable, “good or bad”, in addition to all observable quantities, is one I’ve found very difficult to snap out of in the past, and so I think that framings which make the assumption that this variable exists are probably harmful.
b) There’s no clear line between “benefits from absolute level” and “benefits from relative level”. In the US currency example, a major benefit of a higher absolute level of stability is that it makes your currency more attractive to investors. But of course, when investing, the relative stability of your currency is also crucial. But then again it’s not a zero-sum game because if all currencies become more stable, money will move into them from other asset classes.
The most useful way of understanding this phenomenon is probably something like: “Ceteris parabus, changes in x cause changes in y, and the extent of those changes is greater when x is one of the highest in its reference class.” If I needed to make that pithy, “winner takes more” would probably work well enough.
Quick note that my comments may have come off as confrontational, in which case I apologise; I support people suggesting new terminology in general, and am glad to have had this one brought to my attention.
But the way you solve the St Petersburg paradox in real life is to note that nobody has infinite money, nor infinite time, and therefore it doesn’t matter if your utility function spits out a weird outcome for it because you can have a prior of 0 that it will actually happen. Am I missing something?
“So, does 1+ω make sense (as something different from ω)? It does, for the ordinals and hyperreals only. ”
I am confused because for ordinals, 1+ω = ω. Did you mean ω+1?
Typo: MIT (not AGI) is taking more notice of AGI.
In Defence of Conflict Theory
The thing I usually do, when asked to elicit a probability, is report a probability (usually 2 sig figs) and then also a subjective sense of how easy it would be to shift that probability by giving me more evidence / allowing me more time to think.
What is the correct technical way to summarise the latter quantity (ease of shifting), in an idealised setting?
What we talk about when we talk about maximising utility
I feel like “negotiation” is very handwavey. Can you explain what that looks like in a simple zero-sum situation?
For example, suppose that you can either save the lives of the family of 5 that I described above, or else save 20 loners who have no strong relationships; assume every individual has an equally strong desire to remain alive. How do we actually aggregate all their desires, without the problem of double counting?The reason I think hedonic views are important is because desires can be arbitrarily weird. I don’t want to endorse as moral a parent who raises their child with only one overwhelmingly strong desire—that the sky remains blue. Is that child’s well-being therefore much higher than anyone else’s, since everyone else has had some of their desires thwarted? More generally, I don’t think a “desire” is a particularly well-defined concept, and wouldn’t want it to be my main moral foundation.
I think your first objection is technically correct, but irrelevant to the point I was making; and your second objection is entirely consistent with my conclusion.
On “mistake one”: I am using “I assign utility U to living a happy life” as a shorthand for something like “In general, the difference in utilities I assign between worlds in which I am happily alive, and worlds in which I am not, is U, all else being equal.” This is a perfectly normal sort of phrasing; for example, the wikipedia page on utility says that it “represents satisfaction experienced by the consumer from a good.” Do you object to this and any other talk of utility which isn’t phrased in terms of world-states?
On “mistake two”: I should have mentioned (and will edit to add) that economists don’t endorse interpersonal comparison of economic utility. But I’m not endorsing it either: I’m explicitly flagging it as a philosophical mistake, and explaining one reason why attempts to do so are misguided. This is more useful than simply saying that it’s ill-defined, because the latter leaves us to wonder why we can’t just construct a new way to compare utilities between people—for example, in another comment cousin_it is basically arguing for economic-style utility + interpersonal comparison.
It makes sense, but I find it very counterintuitive, partly because it’s not obvious to me whether the concept of “measuring desire” makes sense. Here are two ways that I might measure whether people have a stronger desire for A or B:
1) I hook up a brainwave reader to each person, and see how strongly/emotional/determined they feel about outcome A vs outcome B.
2) I ask each person whether they would swap outcome A for outcome B.
In the first case, it’s plausible to me that each person’s emotions are basically maxed out at the thought of either their own death, or their family’s death (since we know people are very bad at having emotions which scale appropriately with numbers). So then X = Y, and you save the 20 people.
In the second case, assume that each person involved desires to continue living, personally, at about the same strength S. But then you ask each member of the family whether they’d swap that for someone else in their family surviving, and they’d say yes. So therefore each member of the family has total desire > 5S that their family survives, whereas each loner has desire S to survive themselves, and so you save the family.
Which one is closer to your view of measuring desire? 2 seems more intuitive to me, because it matches the decisions we’d actually make, but then I find the conclusion that it’s more moral to save the family very strange.
Re mistake two:
Okay, so it’s a mistake because it’s simply undefined mathematical nonsense. Now let me define a new form of utility which differs from economic utility only by the fact that interpersonal comparisons are allowed, and occur in whatever way you think is most reasonable. How do you feel about using this new form of utility to draw moral conclusions? I think my arguments are relevant to that question.
Re mistake one:
I’m not assuming that the difference within any pair of world states which differ in a certain way is constant any more than an economist is when they say “let X be the utility that is gained from consuming one unit of good Y”. Both are approximations, but both are useful approximations.
If you’d prefer, I can formalise the situation more precisely terms of world-states. For each world-state, each member of the family assigns it utility equal to the number of family members still alive. So if they all die, that’s 0. If they all survive, that’s 5, and then the total utility from all of them is 25 (assuming we’re working in my “new form of utility” from above, where we can do interpersonal addition).
Meanwhile each loner assigns 1 utility to worlds in which they survive, and 0 otherwise. So now, if we think that maximising utility is moral, we’d say it’s more moral to kill 24 loners than one family of 5, even though each individual values their own life equally. I think that this conclusion is unacceptable, and so it is a reductio of the idea that we should maximise any quantity similar to economic utility.
It seems to me that preference utilitarianism neatly reconciles the general intuitive view against torture with a mathematical utilitarian position. If a proportion p of those 3^^^3 people have a moral compunction against people being tortured, and the remainder are indifferent to torture but have a very slight preference against dust specks, then as long as p is not very small, the overall preference would be for dust specks (and if p was very small, then the moral intuitions of humanity in general have completely changed and we shouldn’t be in a position to make any decisions anyway). Is there something I’m missing?