If you don’t want to violate the independence axiom (which perhaps you did), then you will need bounded utility also when considering deals with non-PEST probabilities.
In any case, if you effectively give probability a lower bound, unbounded utility doesn’t have any specific meaning. The whole point of a double utility is that you will be willing to accept the double utility with half the probability. Once you won’t accept it with half the probability (as will happen in your situation) there is no point in saying that something has twice the utility.
It’s weird, but it’s not quite the same as bounded utility (though it looks pretty similar). In particular, there’s still a point in saying it has double the utility even though you sometimes won’t accept it at half the utility. Note the caveat “sometimes”: at other times, you will accept it.
Suppose event X has utility U(X) = 2 U(Y). Normally, you’ll accept it instead of Y at anything over half the probability. But if you reduce the probabilities of both* events enough, that changes. If you simply had a bound on utility, you would get a different behavior: you’d always accept X and over half the probability of Y for any P(Y), unless the utility of Y was too high. These behaviors are both fairly weird (except in the universe where there’s no possible construction of an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason), but they’re not the same.
Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
So in this case my question is why Kaj suggests his proposal instead of using bounded utility.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.
I think I agree that the OP does not follow independence, but everything else here seems wrong.
Actions A and B are identical except that A gives me 2 utils with .5 probability, while B gives me Graham’s number with .5 probability. I do B. (Likewise if there are ~Graham’s number of alternatives with intermediate payoffs.)
I’m not sure how you thought this was relevant to what I said.
What I was saying was this:
Suppose I say that A has utility 5, and B has utility 10. Basically the statement that B has twice the utility A has, has no particular meaning except that if I would like to have A at a probability of 10%, I would equally like to have B at a probability of 5%. If I would take the 10% chance and not the 5% chance, then there is no longer any meaning to saying that B has “double” the utility of A.
This does totally defy the intuitive understanding of expected utility. Intuitively, you can just set your utility function as whatever you want. If you want to maximize something like saving human lives, you can do that. As in one person dying is exactly half as bad as 2 people dying, which itself is exactly half as bad as 4 people dying, etc.
The justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy. You save the most lives than you would with any other strategy. You lose some people on some bets, but gain many more on other bets.
But this justification and intuitive understanding totally breaks down in the real world. Where there are finite horizons. You don’t get to take an infinite number of bets. An agent following expected utility will just continuously bet away human lives on mugger-like bets, without ever gaining anything. It will always do worse than other strategies.
You can do some tricks to maybe fix this somewhat by modifying the utility function. But that seems wrong. Why are 2 lives not twice as valuable as 1 life. Why are 400 lives not twice as valuable as 200 lives? Will this change the decisions you make in everyday, non-muggle/excessively low probability bets? It seems like it would.
Or we could just keep the intuitive justification for expected utility, and generalize it to work on finite horizons. There are some proposals for methods that do this like mean of quantiles.
I don’t think that “the justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy,” is quite true. Kaj did say something similar to this, but that seems to me a problem with his approach.
Basically, expected utility is supposed to give a mathematical formalization to people’s preferences. But consider this fact: in itself, it does not have any particular sense to say that “I like vanilla ice cream twice as much as chocolate.” It makes sense to say I like it more than chocolate. This means if I am given a choice between vanilla and chocolate, I will choose vanilla. But what on earth does it mean to say that I like it “twice” as much as chocolate? In itself, nothing. We have to define this in order to construct a mathematical analysis of our preferences.
In practice we make this definition by saying that I like vanilla so much that I am indifferent to having chocolate, or to having a 50% chance of having vanilla, and a 50% chance of having nothing.
Perhaps I justify this by saying that it will get me a certain amount of vanilla in my life. But perhaps I don’t—the definition does not justify the preference, it simply says what it means. This means that in order to say I like it twice as much, I have to say that I am indifferent to the 50% bet and to the certain chocolate, no matter what the justification for this might or might not be. If I change my preference when the number of cases goes down, then it will not be mathematically consistent to say that I like it twice as much as chocolate, unless we change the definition of “like it twice as much.”
Basically I think you are mixing up things like “lives”, which can be mathematically quantified in themselves, more or less, and people’s preferences, which only have a quantity if we define one.
It may be possible for Kaj to come up with a new definition for the amount of someone’s preference, but I suspect that it will result in a situation basically the same as keeping our definition, but admitting that people have only a limited amount of preference for things. In other words, they might prefer saving 100,000 lives to saving 10,000 lives, but they certainly do not prefer it 10 times as much, meaning they will not always accept the 100,000 lives saved at a 10% chance, compared to a 100% chance of saving 10,000.
But what on earth does it mean to say that I like it “twice” as much as chocolate?
Obviously it means you would be willing to trade 2 units of chocolate ice cream for 1 unit of vanilla. And over the course of your life, you would prefer to have more vanilla ice cream than chocolate ice cream. Perhaps before you die, you will add up all the ice creams you’ve ever eaten. And you would prefer for that number to be higher rather than lower.
Nowhere in the above description did I talk about probability. And the utility function is already completely defined. I just need to decide on a decision procedure to maximize it.
Expected utility seems like a good choice, because, over the course of my life, different bets I make on ice cream should average themselves out, and I should do better than otherwise. But that might not be true if there are ice cream muggers. Which promise lots of ice cream in exchange for a down payment, but usually lie.
So trying to convince the ice cream maximizer to follow expected utility is a lost cause. They will just end up losing all their ice cream to muggers. They need a system which ignores muggers.
This is definitely not what I mean if I say I like vanilla twice as much as chocolate. I might like it twice as much even though there is no chance that I can ever eat more than one serving of ice cream. If I have the choice of a small serving of vanilla or a triple serving of chocolate, I might still choose the vanilla. That does not mean I like it three times as much.
It is not about “How much ice cream.” It is about “how much wanting”.
I’m saying that the experience of eating chocolate is objectively twice as valuable. Maybe there is a limit on how much ice cream you can eat at a single sitting. But you can still choose to give up eating vanilla today and tomorrow, in exchange for eating chocolate once.
Again, you are assuming there is a quantitative measure over eating chocolate and eating vanilla, and that this determines the measure of my utility. This is not necessarily true, since these are arbitrary examples. I can still value one twice as much as the other, even if they both are experiences that can happen only once in a lifetime, or even only once in the lifetime of the universe.
Sure. But there is still some internal, objective measure of value of experiences. The constraints you add make it harder to determine what they are. But in simple cases, like trading ice cream, it’s easy to determine how much value a person has for a thing.
If you don’t want to violate the independence axiom (which perhaps you did), then you will need bounded utility also when considering deals with non-PEST probabilities.
In any case, if you effectively give probability a lower bound, unbounded utility doesn’t have any specific meaning. The whole point of a double utility is that you will be willing to accept the double utility with half the probability. Once you won’t accept it with half the probability (as will happen in your situation) there is no point in saying that something has twice the utility.
It’s weird, but it’s not quite the same as bounded utility (though it looks pretty similar). In particular, there’s still a point in saying it has double the utility even though you sometimes won’t accept it at half the utility. Note the caveat “sometimes”: at other times, you will accept it.
Suppose event X has utility U(X) = 2 U(Y). Normally, you’ll accept it instead of Y at anything over half the probability. But if you reduce the probabilities of both* events enough, that changes. If you simply had a bound on utility, you would get a different behavior: you’d always accept X and over half the probability of Y for any P(Y), unless the utility of Y was too high. These behaviors are both fairly weird (except in the universe where there’s no possible construction of an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason), but they’re not the same.
Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.
I think I agree that the OP does not follow independence, but everything else here seems wrong.
Actions A and B are identical except that A gives me 2 utils with .5 probability, while B gives me Graham’s number with .5 probability. I do B. (Likewise if there are ~Graham’s number of alternatives with intermediate payoffs.)
I’m not sure how you thought this was relevant to what I said.
What I was saying was this:
Suppose I say that A has utility 5, and B has utility 10. Basically the statement that B has twice the utility A has, has no particular meaning except that if I would like to have A at a probability of 10%, I would equally like to have B at a probability of 5%. If I would take the 10% chance and not the 5% chance, then there is no longer any meaning to saying that B has “double” the utility of A.
This does totally defy the intuitive understanding of expected utility. Intuitively, you can just set your utility function as whatever you want. If you want to maximize something like saving human lives, you can do that. As in one person dying is exactly half as bad as 2 people dying, which itself is exactly half as bad as 4 people dying, etc.
The justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy. You save the most lives than you would with any other strategy. You lose some people on some bets, but gain many more on other bets.
But this justification and intuitive understanding totally breaks down in the real world. Where there are finite horizons. You don’t get to take an infinite number of bets. An agent following expected utility will just continuously bet away human lives on mugger-like bets, without ever gaining anything. It will always do worse than other strategies.
You can do some tricks to maybe fix this somewhat by modifying the utility function. But that seems wrong. Why are 2 lives not twice as valuable as 1 life. Why are 400 lives not twice as valuable as 200 lives? Will this change the decisions you make in everyday, non-muggle/excessively low probability bets? It seems like it would.
Or we could just keep the intuitive justification for expected utility, and generalize it to work on finite horizons. There are some proposals for methods that do this like mean of quantiles.
I don’t think that “the justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy,” is quite true. Kaj did say something similar to this, but that seems to me a problem with his approach.
Basically, expected utility is supposed to give a mathematical formalization to people’s preferences. But consider this fact: in itself, it does not have any particular sense to say that “I like vanilla ice cream twice as much as chocolate.” It makes sense to say I like it more than chocolate. This means if I am given a choice between vanilla and chocolate, I will choose vanilla. But what on earth does it mean to say that I like it “twice” as much as chocolate? In itself, nothing. We have to define this in order to construct a mathematical analysis of our preferences.
In practice we make this definition by saying that I like vanilla so much that I am indifferent to having chocolate, or to having a 50% chance of having vanilla, and a 50% chance of having nothing.
Perhaps I justify this by saying that it will get me a certain amount of vanilla in my life. But perhaps I don’t—the definition does not justify the preference, it simply says what it means. This means that in order to say I like it twice as much, I have to say that I am indifferent to the 50% bet and to the certain chocolate, no matter what the justification for this might or might not be. If I change my preference when the number of cases goes down, then it will not be mathematically consistent to say that I like it twice as much as chocolate, unless we change the definition of “like it twice as much.”
Basically I think you are mixing up things like “lives”, which can be mathematically quantified in themselves, more or less, and people’s preferences, which only have a quantity if we define one.
It may be possible for Kaj to come up with a new definition for the amount of someone’s preference, but I suspect that it will result in a situation basically the same as keeping our definition, but admitting that people have only a limited amount of preference for things. In other words, they might prefer saving 100,000 lives to saving 10,000 lives, but they certainly do not prefer it 10 times as much, meaning they will not always accept the 100,000 lives saved at a 10% chance, compared to a 100% chance of saving 10,000.
Obviously it means you would be willing to trade 2 units of chocolate ice cream for 1 unit of vanilla. And over the course of your life, you would prefer to have more vanilla ice cream than chocolate ice cream. Perhaps before you die, you will add up all the ice creams you’ve ever eaten. And you would prefer for that number to be higher rather than lower.
Nowhere in the above description did I talk about probability. And the utility function is already completely defined. I just need to decide on a decision procedure to maximize it.
Expected utility seems like a good choice, because, over the course of my life, different bets I make on ice cream should average themselves out, and I should do better than otherwise. But that might not be true if there are ice cream muggers. Which promise lots of ice cream in exchange for a down payment, but usually lie.
So trying to convince the ice cream maximizer to follow expected utility is a lost cause. They will just end up losing all their ice cream to muggers. They need a system which ignores muggers.
This is definitely not what I mean if I say I like vanilla twice as much as chocolate. I might like it twice as much even though there is no chance that I can ever eat more than one serving of ice cream. If I have the choice of a small serving of vanilla or a triple serving of chocolate, I might still choose the vanilla. That does not mean I like it three times as much.
It is not about “How much ice cream.” It is about “how much wanting”.
I’m saying that the experience of eating chocolate is objectively twice as valuable. Maybe there is a limit on how much ice cream you can eat at a single sitting. But you can still choose to give up eating vanilla today and tomorrow, in exchange for eating chocolate once.
Again, you are assuming there is a quantitative measure over eating chocolate and eating vanilla, and that this determines the measure of my utility. This is not necessarily true, since these are arbitrary examples. I can still value one twice as much as the other, even if they both are experiences that can happen only once in a lifetime, or even only once in the lifetime of the universe.
Sure. But there is still some internal, objective measure of value of experiences. The constraints you add make it harder to determine what they are. But in simple cases, like trading ice cream, it’s easy to determine how much value a person has for a thing.