I don’t understand your argument that the median utility maximizer would buckle its seat belt in the real world. It seemed kind of like you might be trying to argue that median utility maximizers and expected utility maximizers would always approximate each other under realistic conditions, but since you then argue that the alleged difference in their behavior on the Pascal’s mugging problem is a reason to prefer median utility maximizers (implying that Pascal’s mugging-type problems should be accepted as realistic, or at least that getting them correct is important in a way that getting “buckle my seatbelt, given that this is the only decision I will ever make” right isn’t), so I guess that’s not it.
But anyway, even if you are right that median utility maximizers buckle their seatbelts in the context of a realistic collections of choices, you concede that they do not buckle their seatbelts when the decision is isolated, and that this is the incorrect decision. I think you should take the fact that your proposal gets a really easy problem wrong much more seriously. If it can’t get the seatbelt problem right, it is a bad algorithm, and bad algorithms should not be expected to perform well in real-world problems. I would give an example of a real-world problem that it performs poorly on, but I would have said something like the seatbelt problem, and since I don’t understand your argument that it gets that right in the real world, I don’t know what must be done in order to construct an example to which your argument does not apply.
Furthermore, I am unimpressed that median utility maximizers reject Pascal’s mugging. If you take a random function from decision problems to decisions, there is about a 50% chance it will reject Pascal’s mugging, but that doesn’t make it a good decision theory. And median utility maximizers do not reject Pascal’s mugging for correct reasons. To see this, note that if the seatbelt problem is considered in isolation, it looks exactly like the Pascal’s mugging problem, in terms of all the information that median utility maximizers pay attention to, so median utility maximizers do analogous actions in each problem (don’t bother putting your seatbelt on, and don’t pay the mugger, respectively). However, there are important differences between the problems that make it correct to put your seatbelt on but not pay the mugger. Since a median utility maximizer does not consider these differences, its decision not to pay the mugger does not take into account the reasons that it is a good idea not to pay the mugger. It appears to me that you are not even really trying to come up with a way to make the right decisions for the right reasons, and instead you are merely trying to find a way to make the right decisions. I think that this approach is misguided, because the space of possible failure modes for a decision theory is vast, so if you successfully kludge together a decision procedure into performing well on a certain reasonably finite collection of decision problems, without ensuring that it arrives at its decisions in ways that make sense, the chances that it performs well on all decision problems, or even most of them, is vanishingly small.
Since you brought up the iterated Pascal’s mugging, perhaps part of your motivation for this was to find something that would not pay in the isolated Pascal’s mugging, but pay each time in the iterated Pascal’s mugging? First of all, as literally stated, paying each time in the iterated Pascal’s mugging isn’t even an available option (I don’t have $5 billion, so I can’t pay off 1 billion muggers), so it is trivially false that the correct action is to pay every time. However, it is true that there are interpretations of what you could mean under which I would agree that paying is the correct action. But in those cases, an expected utility maximizer with a reasonable bounded utility function will pay, even while not paying in the standard Pascal’s mugging problem. (The naive model of the situation in which iterating the problem does not change how an expected utility maximizer handles it does not correctly model the interpretation of “iterated Pascal’s mugging” in which it makes sense to pay. I’d say what I mean, but actually keeping track of everything relevant to the problem makes it somewhat tedious to explain.)
I don’t understand your argument that the median utility maximizer would buckle its seat belt in the real world.
It derives from the fact that median maximalisation doesn’t consider decisions independently, even if their gains and losses are independent.
For illustration, compare the following deal: you pay £q, and get £1 with probability p. There are n independent deals (assume your utility is linear in £).
If n=1, the median maximiser accepts the deal iff q0.5. Not a very good performance! Now let’s look at larger n. For m < n, accepting m deals gets you an expected reward of m(p-q). The median is a bit more complicated (see https://en.wikipedia.org/wiki/Binomial_distribution#Mode_and_median ), but it’s within £1 of the mean reward.
So if pq, it will accept all n deals.
For pq, it will accept at least n − 1/(p-q) deals. In all cases, its expected loss, compared with the mean maximiser, is less than £1.
There’s a similar effect going on when considering the seat-belt situation. Aggregation concentrates the distribution in a way that moved median and mean towards each other.
You appear to now be making an argument that you already conceded was incorrect in OP:
This means that the decision of a median maximiser will be close to those of a utility maximiser—they take almost the same precautions—though the outcomes are still pretty far apart: the median maximiser accepts a 49.99999...% chance of death.
You then go on to say that if the agent also faces many decisions of a different nature, it won’t do that. That’s where I get lost.
The median maximiser accepts a 49.99999...% chance of death, only because “death”, “trivial cost” and “no cost” are the only options here. If I add “severe injury” and “light injury” to the outcomes, the maximiser will now accept less than a 49.9999...% chance of light injury. If we make light injury additive, and make the trivial cost also additive and not incomparable to light injuries, we get something closer to my illustrative example above.
Suppose it comes up with 2 possible policies, one of which involves a 49% chance of death and no chance of injury, and another which involves a 49% chance of light injury, and no chance of heavy injury or death. The median maximizer sees no reason to prefer the second policy if they have the same effects the other 51% of the time.
Er, yes, constructing single choice examples when the median behaves oddly/wrongly is trivial. My whole point is about what happens to median when you aggregate decisions.
You were claiming that in a situation where a median-maximizing agent has a large number of trivially inconvenient action that prevent small risks of death, heavy injury, or light injury, then it would accept a 49% chance of light injury, but you seemed to imply that it would not accept a 49% chance of death. I was trying to point out that this appears to be incorrect.
I’m not entirely sure what your objection is; we seem to be talking at cross purposes.
Let’s try it simpler. If we assume that the cost of buckling seat belts is incommensurable (in practice) with light injury (and heavy injury, and death), then the median maximising agent will accept a 49.99..% chance of (light injury or heavy injury or death), over their lifetime. Since light injury is much more likely than death, this in effect forces the probability of death down to a very low amount.
It’s just an illustration of the general point that median maximising seems to perform much better in real-world problems than its failure in simple theoretical ones would suggest.
Since light injury is much more likely than death, this in effect forces the probability of death down to a very low amount.
No, it doesn’t. That does not address the fact that the agent will not preferentially accept light injury over death. Adopting a policy of immediately committing suicide once you’ve been injured enough to force you into the bottom half of outcomes does not decrease median utility. The agent has no incentive to prevent further damage once it is in the bottom half of outcomes. As a less extreme example, the value of house insurance to a median maximizer is 0, just because loosing your house is a bad outcome even if you get insurance money for it. This isn’t a weird hypothetical that relies on it being an isolated decision; it’s a real-life decision that a median maximizer would get wrong.
A more general way of stating how multiple decisions improve median maximalisation: the median maximaliser is indifferent of outcomes not at the median (eg suicide vs light injury). But as the decision tree grows and the number of possible situations does as well, the probability increases that outcomes not at the median in a one shot, will affect the median in the more complex situation.
This argument relies on your utility being a sum of effects from each of the decisions you made, but in reality, your decisions interact in much more complicated ways, so that isn’t a realistic model.
Also, if your defense of median maximization consists entirely of an argument that it approximates mean maximization, then what’s the point of all this? Why not just use expected utility maximization? I’m expecting you to bring up Pascal’s mugging here, but since VNM-rationality does not force you to pay the mugger, you’ll have to do better than that.
This argument relies on your utility being a sum of effects from each of the decisions you made
It doesn’t require that in the least. I don’t know if, eg, quadratic of higher order effects would improve or worsen the situation.
but since VNM-rationality does not force you to pay the mugger
The consensus at the moment seems to be that if you have unbounded utility, it does force you to pay some muggers. Now, I’m perfectly fine with bounding your utility to avoid muggers, but that’s the kind of non-independent decision some people don’t like ;-)
The real problem is things like the Cauchy distribution, or any function without an expectation value at all. Saying “VNM works fine as long as we don’t face these difficult choices, then it breaks down” is very unsatisfactory. I’m also interested in seeing what happens when “expect to win” and “win in expectation” become quite distinct—a rare event, in practice.
That example also relies on your utility being the sum of components that are determined from your various actions.
Choosing to bound an unbounded utility function to avoid muggers does not.
To be clear, I was not suggesting that you have an unbounded utility function that it would make sense for you to maximize if it weren’t for Pascal’s mugger, so you should bound it when there might be a Pascal’s mugger around. I was suggesting that the utility function it makes sense for you to maximize is bounded. Unbounded utility functions are so loony they never should have been seriously considered in the first place; Pascal’s mugger is merely a dramatic illustration of that fact.
Edit: I probably shouldn’t rely on the theoretical reasons to prefer bounded utility functions, since they are not completely airtight and actual human preferences are more important anyway. So let’s look at actual human preferences. Suppose you’ve got a rational agent with preference relation “<”, and you want to test whether its utility function is bounded or unbounded. Here’s a simple test: First find outcomes A and B such that A<B (if you can’t even do that, its utility function is constant, hence bounded). Then pick an absurdly tiny probability p>0. Now see if you can find such a terrible C and such a wonderful D that pC+(1-p)B < pD + (1-p)A. If, for every p>0 you can find such C and D, then its utility function is unbounded. But if for some p>0, you cannot find any C and D that will suffice, even when you probe the extremes of goodness and badness, then its utility function is bounded. This test should sound familiar. What I’m getting at here is that one does not bound their unbounded utility function so that they don’t have to pay Pascal’s mugger; your preferences were simply bounded all along, and your response to Pascal’s mugger is proof.
Look, we’re arguing past each other here. My logical response here would be to add more options to the system, which would remove the problem you identified (and I don’t understand your house insurance example—this is just the seat-belt decision again as a one-shot, and I would address it by looking at all the financial decisions you make in your life—and if that’s not enough, all the decisions, including all the “don’t do something clearly stupid and pointless” ones).
What I think is clear is:
a) Median maximalisation makes bad decisions in isolated problems.
b) If we combine all the likely decisions that a median maximiser will have to make, the quality of the decisions increase.
If you want to argue against it, either say that a) is bad enough we should reject the approach anyway, even if it decides well in practice, or find examples where a real world median maximaliser will make bad decisions even in the real world (if you would pay Pascal’s mugger, then you could use that as an example).
I don’t understand your house insurance example—this is just the seat-belt decision again as a one-shot
We were modeling the seat-belt decision as something that makes the difference between being dead and being completely fine in the event of an accident (which I suppose is not very realistic, but whatever). I was trying to point to a situation where an event can happen which is bad enough to put in the bottom half of outcomes either way, so that nothing that happens conditional on the event can affect the median outcome, but a decision you can make ahead of time would make the difference between bad and worse.
I do think that a) is bad enough, because a decision procedure that does poorly in isolated problems is wrong, and thus cannot be expected to do well in realistic situations, as I mentioned previously. I guess b) is probably technically true, but it is not enough for the quality of the decisions to increase when the number increases; it should actually increase towards a limit that isn’t still awful, and come close to achieving that limit (I’m pretty sure it fails on at least one of those, though which step it fails on might depend on how you make things precise). I’ve given examples where median maximizers make bad decisions in the real world, but you’ve dismissed them with vague appeals to “everything will be fine when you consider it in the context of all the other decisions it has to make”.
I’ve given examples where median maximizers make bad decisions in the real world, but you’ve dismissed them with vague appeals to “everything will be fine when you consider it in the context of all the other decisions it has to make”.
And I’ve added in the specific other decisions needed to achieve this effect. I agree it’s not clear what exactly the median maximalisation converge on in the real world, but the examples you’ve produced are not sufficient to show it’s bad.
I do think that a) is bad enough, because a decision procedure that does poorly in isolated problems is wrong
My take on this is that counterfactual decision count as well. ie if humans look not only at the decisions they face, but the ones they can imagine facing, then median maximalisation is improved. My justification for this line of thought is—how do you know that one chocolate cake is +10 utility while one coffee is +2 (and two coffees is +3, three is +2, and four is −1)? Not just the ordinal ranking, but the cardinality. I’d argue that you get this by either experiencing circumstances where you choose a 20% chance of a cake over coffee, or imagining yourself in that circumstance. And if imagination and past experiences are valid for the purpose of constructing your utility function, they should be valid for the purpose of median-maximalisation.
And I’ve added in the specific other decisions needed to achieve this effect.
That you claim achieve that effect. But as I said, unless the are choices you can make that would protect you from light injury involve less inconvenience per % reduction in risk than the choices you can make that would protect you from death, it doesn’t work.
However, I did think of something which seems to sort of achieve what you want: if you have high uncertainty about what the value of your utility function will be, then adding something to it with some probability will have a significant effect on the median value, even if the probability is significantly less than 50%. For instance, a 49% chance of death is very bad because if there’s a 49% chance you die, then the median outcome is one in which you’re alive but in a worse situation than all but 1⁄51 of the scenarios in which you die. It may be that this is what you had in mind, and adding future decisions that involve uncertainty was merely a mechanism by which large uncertainty about the outcome was introduced, in which case future-you actually getting to make any choices about them was a red herring. I still don’t find this argument convincing either, though, both because it still undervalues protection from risks of losses that are large relative to the rest your uncertainty about the value of the outcome (for instance, note that when valuing reductions in risk of death, there is still a weird discontinuity around 50%), and because it assumes that you can’t make decisions that selectively have significant consequences only in very good or very bad outcomes (this is what I was getting at with the house insurance example).
My take on this is that counterfactual decision count as well. … And if imagination and past experiences are valid for the purpose of constructing your utility function, they should be valid for the purpose of median-maximalisation.
I don’t understand what you’re saying here. Is it that you can maximize the median value of the mean of the values of your utility function in a bunch of hypothetical scenarios? If so, that sounds kind of like Houshalter’s median of means proposal, which approaches mean maximization as the number of samples considered approaches infinity.
The observation I have is that when facing many decisions, median maximialisation tends to move close to mean maximalisation (since the central limit theorem has “convergence in the distribution”, the median will converge to the mean in the case of averaging repeated independent processes; but there are many other examples of this). Therefore I’m considering what happens if you add “all the decisions you can imagine making” to the set of actual decisions you expect to make. This feels like it should move the two even closer together.
Ah, are you saying you should use your prior to choose a policy that maximizes your median utility, and then implementing that policy, rather than updating your prior with your observations and then choosing a policy that maximizes the median? So like UDT but with medians?
It seems difficult to analyze how it would actually behave, but it seems likely to be true that it acts much more similarly to mean utility maximization than it would if you updated before choosing the policy. Both of these properties (difficulty to analyze, and similarity to mean maximization) make it difficult to identify problems that it would perform poorly on. But this also makes it difficult to defend its alleged advantages (for instance, if it ends up being too similar to mean maximization, and if you use an unbounded utility function as you seem to insist, perhaps it pays Pascal’s mugger).
Ah, are you saying you should use your prior to choose a policy that maximizes your median utility, and then implementing that policy, rather than updating your prior with your observations and then choosing a policy that maximizes the median? So like UDT but with medians?
Ouch! Sorry for not being clear. If you missed that, then you can’t have understood much of what I was saying!
How do you know that it’s right to buckle your seatbelt? If you are only going to ride in a car once, never again. And there are no other risks to your life, and so no need to make a general policy against taking small risks?
I’m not confident that it’s actually the wrong choice. And if it is, it shouldn’t matter much. 99.99% of the time, the median will come out with higher utility than the EU maximizer.
This is generalizable. If there was a “utility competition” between different decision policies in the same situations, the median utility would usually come out on top. As the possible outcomes become more extreme and unlikely, expected utility will do worse and worse. With pascal’s mugging at the extreme.
That’s because EU trades away utility from the majority of possible outcomes, to really really unlikely outcomes. Outliers can really skew the mean of a distribution, and EU is just the mean.
Of course median can be exploited too. Perhaps there is some compromise between them that gets the behavior we want. There are an infinite number of possible policies for deciding which distribution of utilities to prefer.
EU was chosen because it is the only one that meets a certain set of conditions and is perfectly consistent. But if you allow for algorithms that select overall policies instead of decisions, like OP does, then you can make many different algorithms consistent.
So there is no inherent reason to prefer mean over median. It just comes down to personal preference, and subjective values. What probability distribution of utilities do you prefer?
How do you know that it’s right to buckle your seatbelt? If you are only going to ride in a car once, never again.
I do think that the isolation of the decision is a red herring, but for the sake of the point I was trying to make, it is probably easier to replace the example with a structurally similar one in which the right answer is obvious: suppose you have the opportunity to press a button that will kill you will 49% probability, and give you $5 otherwise. This is the only decision you will ever make. Should you press the button?
Perhaps there is some compromise between them that gets the behavior we want.
As I was saying in my previous comment, I think that’s the wrong approach. It isn’t enough to kludge together a decision procedure that does what you want on the problems you thought of, because then it will do something you don’t want on something you haven’t thought of. You need a decision procedure that will reliably do the right thing, and in order to get that, you need it to do the right thing for the right reasons. EU maximization, applied properly, will tell you to do the correct things, and will do so for the correct reasons.
So there is no inherent reason to prefer mean over median.
Please reread the OP and my comment. If you allow selection over policies instead of individual decisions, you can be perfectly consistent. EU and median are both special cases of ways to pick policies, based on the probability distribution of utility they produce.
You need a decision procedure that will reliably do the right thing, and in order to get that, you need it to do the right thing for the right reasons. EU maximization, applied properly, will tell you to do the correct things, and will do so for the correct reasons.
There is no law of the universe that some procedures are correct and others aren’t. You just have to pick one that you like, and your choice is going to be arbitrary.
If you go with EU you are pascal muggable. If you go with median you are muggable in certain cases as well (though you should usually, with >50% probability, end up with better outcomes in the long run. Whereas EU could possibly fail 100% of the time. So it’s exploitable, but it’s less exploitable at least.)
If you allow selection over policies instead of individual decisions, you can be perfectly consistent.
I don’t see how selecting policies instead of actions removes the motivation for independence.
You just have to pick one that you like, and your choice is going to be arbitrary.
Ultimately, it isn’t the policy that you care about; it’s the outcome. So you should pick a policy because you like the probability distributions over outcomes that you get from implementing it more than you like the probability distributions over outcomes that you would get from implementing other policies. Since there are many decision problems to use your policy on, this quite heavily constrains what policy you choose. In order to get a policy that reliably picks the actions that you decide are correct in the situations where you can tell what the correct action is, it will have to make those decisions for the same reason you decided that it was the best action (or at least something equivalent to or approximating the same reason). So no, the choice of policy is not at all arbitrary.
If you go with EU you are pascal muggable.
That is not true. EU maximizers with bounded utility functions reject Pascal’s wager.
I don’t see how selecting policies instead of actions removes the motivation for independence.
There are two reasons to like independence. First of all, you might like it for philosophical/aesthetic reasons: “these things really should be independent, these really should be irrelevant”. Or you could like it because it prevents you from being money pumped.
When considering policies, money pumping is (almost) no longer an issue, because a policy that allows itself to be money-pumped is (almost) certainly inferior to one that doesn’t. So choosing policies removes one of the motivations for independence, to my mind the important one.
While it’s true that this does not tell you to pay each time to switch the outcomes around in a circle over and over again, it still falls prey to one step of a similar problem. Suppose their are 3 possible outcomes: A, B, and C, and there are 2 possible scenarios: X and Y. In scenario X, you get to choose between A and B. In scenario Y, you can attempt to choose between A and B, and you get what you picked with 50% probability, and you get outcome C otherwise. In each scenario, this is the only decision you will ever make. Suppose in scenario X, you prefer A over B, but in scenario Y, you prefer (B+C)/2 over (A+C)/2. But suppose you had to pay to pick A in scenario X, and you had to pay to pick (B+C)/2 in scenario Y, and you still make those choices. If Y is twice as likely as X a priori, then you are paying to get a probability distribution over outcomes that you could have gotten for free by picking B given X, and (A+C)/2 given Y. Since each scenario only involves you ever getting to make one decision, picking a policy is equivalent to picking a decision.
Your example is difficult to follow, but I think you are missing the point. If there is only one decision, then it’s actions can’t be inconsistent. By choosing a policy only once—one that maximizes it’s desired probability distribution of utility outcomes—it’s not money pumpable, and it’s not inconsistent.
Now by itself it still sucks because we probably don’t want to maximize for the best median future. But it opens up the door to more general policies for making decisions. You no longer have to use expected utility if you want to be consistent. You can choose a tradeoff between expected utility and median utility (see my top level comment), or a different algorithm entirely.
If there is only one decision point in each possible world, then it is impossible to demonstrate inconsistency within a world, but you can still be inconsistent between different possible worlds.
Edit: as V_V pointed out, the VNM framework was designed to handle isolated decisions. So if you think that considering an isolated decision rather than multiple decisions removes the motivation for the independence axiom, then you have misunderstood the motivation for the independence axiom.
So if you think that considering an isolated decision rather than multiple decisions removes the motivation for the independence axiom, then you have misunderstood the motivation for the independence axiom.
I understand the two motivations for the independence axiom, and the practical one (“you can’t be money pumped”) is much more important that the theoretical one (“your system obeys this here philosophically neat understanding of irrelevant information”).
But this is kind of a moot point, because humans don’t have utility functions. And therefore we will have to construct them. And the process of constructing them is almost certainly going to depend on facts about the world, making the construction process almost certainly inconsistent between different possible worlds.
And the process of constructing them is almost certainly going to depend on facts about the world
It shouldn’t. If your preferences among outcomes depend on what options are actually available to you, then I don’t see how you can justify claiming to have preferences among outcomes, as opposed to tendencies to make certain choices.
Then define me a process that takes people’s current mess of preferences, makes these into utility functions, and, respecting bounded rationality, is independent of options available in the real world. Even then, we have the problem that this mess of preferences is highly dependent on real world experiences in the first place.
I don’t see how you can justify claiming to have preferences among outcomes, as opposed to tendencies to make certain choices.
If I always go left at a road, I have tendency to make certain choices. If I have a full model of the entire universe with labelled outcomes ranked on a utility function, and use it with unbounded rationality to make decisions, I have preferences among outcomes. The extremes are clear.
I feel that a bounded human being with a crude mental model that is trying to achieve some goal, imperfectly (because of ingrained bad habits, for instance) is better described as having preferences among outcomes. You could argue that they have mere tendencies, but this seems to stretch the term. But in any case, this is a simple linguistic dispute. Real human beings cannot achieve independence.
Then define me a process that takes people’s current mess of preferences, makes these into utility functions, and, respecting bounded rationality, is independent of options available in the real world.
Define me a process with all those properties except the last one. If you can’t do that either, it’s not the last constraint that is to blame for the difficulty.
Even then, we have the problem that this mess of preferences is highly dependent on real world experiences in the first place.
Yes, different agents have different preferences. The same agent shouldn’t have its preferences change when the available outcomes do.
If I have a full model of the entire universe with labelled outcomes ranked on a utility function, and use it with unbounded rationality to make decisions, I have preferences among outcomes.
If you are neutral between .4A+.6C and .4B+.6C, then you don’t have a very good claim to preferring A over B.
Define me a process with all those properties except the last one.
Well, there’s my old idea here: http://lesswrong.com/lw/8qb/cevinspired_models/ . I don’t think it’s particularly good, but it does construct a utility function, and might be doable with good enough models or a WBE. More broadly, there’s the general “figure out human preferences from their decisions and from hypothetical questions and fit a utility function to it”, which we can already do today (see “inverse reinforcement learning”); we just can’t do it well enough, yet, to get something generally safe at the other end.
None of these ideas have independent variants (not technically true; I can think of some independent versions of them, but they’re so ludicrously unsafe in our world that we’d rule them out immediately; thus, this would be a non-independent process).
If you are neutral between .4A+.6C and .4B+.6C, then you don’t have a very good claim to preferring A over B.
?
If I actually do prefer A over B (and my behaviour reflects that in (1- ɛ)A+ ɛC versus (1-ɛ)B+ ɛC cases), then I have an extremely good claim to preferring A over B, and an extremely poor claim to independence.
I assumed accuracy was implied by “making a mess of preferences into a utility function”.
More broadly, there’s the general “figure out human preferences from their decisions and from hypothetical questions and fit a utility function to it”, which we can already do today (see “inverse reinforcement learning”); we just can’t do it well enough, yet, to get something generally safe at the other end.
I’m somewhat skeptical of that strategy for learning utility functions, because the space of possible outcomes is extremely high-dimensional, and it may be difficult to test extreme outcomes because the humans you’re trying to construct a utility function for might not be able to understand them. But perhaps this objection doesn’t get to the heart of the matter, and I should put it aside for now.
None of these ideas have independent variants
I am admittedly not well-versed in inverse reinforcement learning, but this is a perplexing claim. Except for a few people like you suggesting alternatives, I’ve only ever heard “utility function” used to refer to a function you maximize the expected value of (if you’re trying to handle uncertainty), or a function you just maximize the value of (if you’re not trying to handle uncertainty). In the first case, we have independence. In the second case, the question of whether or not we obey independence doesn’t really make sense. So if inverse reinforcement learning violates independence, then what exactly does it try to fit to human preferences?
If I actually do prefer A over B
Then if the only difference between two gambles is that one might give you A when the other might give you B, you’ll take the one that might give you something you like instead of something you don’t like.
I’ve only ever heard “utility function” used to refer to
To be clear, I am saying the process of constructing the utility function violates independence, not that subsequently maximising it does. Similarly, choosing a median-maximising policy P violates independence, but there is (almost certainly) a utility u such that maximising u is the same as following P.
Once the first choice is made, we have independence in both cases; before it is made, we have it in neither. The philosophical underpinning of independence in single decisions therefore seems very weak.
To be clear, I am saying the process of constructing the utility function violates independence
Feel free to tell me to shut up and learn how inverse reinforcement learning works before bothering you with such questions, if that is appropriate, but I’m not sure what you mean. Can you be more precise about what property you’re saying inverse reinforcement learning doesn’t have?
Inverse reinforcement learning relies on observation of humans performing specific actions, and drawing the “right” conclusion as to what their preferences. Indirectly, it relies on humans tinkering with its code to remove “errors”, ie things that don’t fit with the mental image that human programmers of what preferences should be.
Given that human desires are not independent (citation not needed), this process, if it produces a utility function, involves constructing something independent from non-independent input. However, to establish this utility function, the algorithm has access only to the particular problems given to it, and the particular mental images of its programmers. It is almost certain that the end result would be somewhat different if it was trained on different problems, or if its programmers had different intuitions. Therefore the process itself cannot be independent.
Ah, I see what you mean, and you’re right; the utility function constructed will depend on how the data points are sampled. This isn’t quite the same as saying that the result will depend on what results are actually available, though, unless knowledge about what results will be available is used to determine how to sample the data. This still seems like somewhat of a defect of inverse reinforcement learning, unless there ends up being a good case that some particular way of sampling the data is optimal for revealing underlying preferences and ignoring biases, or something like that.
Given that human desires are not independent (citation not needed)
That’s probably true, but on the other hand, you seem to want to pin the deviations of human behavior from VNM rationality on violations of the independence axiom, and it isn’t clear to me that this is the case (I don’t think the point you were making relies on this, so if you weren’t trying to make that claim then you can ignore this; it just seemed like you might be). There are situations where there are large framing effects (that is, whether A or B is preferred depends on how the options are presented, even if no other outcome C is being mixed in with them), and likely also violations of transitivity (where someone would say A>B, B>C, and C>A whenever you ask them about 2 of them without bringing up the third). It seems likely to me that most paradoxes of human decision-making have more to do with these than they do to violations of independence.
It can’t be inconsistent within a world no matter how many decisions points there are. If we agree it’s not inconsistent, then what are you arguing against?
I don’t care about the VNM framework. As you said, it is designed to be optimal for decisions made in isolation. Because we don’t need to make decisions in isolation, we don’t need to be constrained by it.
No. Inconsistency between different possible worlds is still inconsistency.
Because we don’t need to make decisions in isolation, we don’t need to be constrained by it.
The difference doesn’t matter that much in practice. If there are multiple decision points, you can combine them into one by selecting a policy, or by considering them sequentially and using your beliefs about what your choices will be in the future to compute the expected utilities of the possible decisions available to you now. The reason that the VNM framework was designed for one-shot decisions is that it makes things simpler without actually constraining what it can be applied to.
No. Inconsistency between different possible worlds is still inconsistency.
It’s perfectly consistent in the sense that it’s not money pumpable, and always makes the same decisions given the same information. It will make different decisions in different situations, given different information. But that is not inconsistent by an reasonable definition of “inconsistent”.
The difference doesn’t matter that much in practice.
It makes a huge difference. If you want to get the best median future, then you can’t make decisions in isolation. You need to consider every possible decision you will have to make, and their probability. And choose a decision policy that selects the best median outcome.
It’s perfectly consistent in the sense that it’s not money pumpable, and always makes the same decisions given the same information.
As in my previous example (sorry about it being difficult to follow, though I’m not sure yet what I could say to clarify things), it is inconsistent in the sense that it can lead you to pay for probability distributions over outcomes that you could have achieved for free.
You need to consider every possible decision you will have to make, and their probability.
Right. As I just said, “you can… consider them sequentially and use your beliefs about what your choices will be in the future to compute the expected utilities of the possible decisions available to you now.” (edited to fix grammar). This reduces iterated decisions to isolated decisions: you have certain beliefs about what you’ll do in the future, and now you just have to make a decision on the issue facing you now.
I don’t understand your argument that the median utility maximizer would buckle its seat belt in the real world. It seemed kind of like you might be trying to argue that median utility maximizers and expected utility maximizers would always approximate each other under realistic conditions, but since you then argue that the alleged difference in their behavior on the Pascal’s mugging problem is a reason to prefer median utility maximizers (implying that Pascal’s mugging-type problems should be accepted as realistic, or at least that getting them correct is important in a way that getting “buckle my seatbelt, given that this is the only decision I will ever make” right isn’t), so I guess that’s not it.
But anyway, even if you are right that median utility maximizers buckle their seatbelts in the context of a realistic collections of choices, you concede that they do not buckle their seatbelts when the decision is isolated, and that this is the incorrect decision. I think you should take the fact that your proposal gets a really easy problem wrong much more seriously. If it can’t get the seatbelt problem right, it is a bad algorithm, and bad algorithms should not be expected to perform well in real-world problems. I would give an example of a real-world problem that it performs poorly on, but I would have said something like the seatbelt problem, and since I don’t understand your argument that it gets that right in the real world, I don’t know what must be done in order to construct an example to which your argument does not apply.
Furthermore, I am unimpressed that median utility maximizers reject Pascal’s mugging. If you take a random function from decision problems to decisions, there is about a 50% chance it will reject Pascal’s mugging, but that doesn’t make it a good decision theory. And median utility maximizers do not reject Pascal’s mugging for correct reasons. To see this, note that if the seatbelt problem is considered in isolation, it looks exactly like the Pascal’s mugging problem, in terms of all the information that median utility maximizers pay attention to, so median utility maximizers do analogous actions in each problem (don’t bother putting your seatbelt on, and don’t pay the mugger, respectively). However, there are important differences between the problems that make it correct to put your seatbelt on but not pay the mugger. Since a median utility maximizer does not consider these differences, its decision not to pay the mugger does not take into account the reasons that it is a good idea not to pay the mugger. It appears to me that you are not even really trying to come up with a way to make the right decisions for the right reasons, and instead you are merely trying to find a way to make the right decisions. I think that this approach is misguided, because the space of possible failure modes for a decision theory is vast, so if you successfully kludge together a decision procedure into performing well on a certain reasonably finite collection of decision problems, without ensuring that it arrives at its decisions in ways that make sense, the chances that it performs well on all decision problems, or even most of them, is vanishingly small.
Since you brought up the iterated Pascal’s mugging, perhaps part of your motivation for this was to find something that would not pay in the isolated Pascal’s mugging, but pay each time in the iterated Pascal’s mugging? First of all, as literally stated, paying each time in the iterated Pascal’s mugging isn’t even an available option (I don’t have $5 billion, so I can’t pay off 1 billion muggers), so it is trivially false that the correct action is to pay every time. However, it is true that there are interpretations of what you could mean under which I would agree that paying is the correct action. But in those cases, an expected utility maximizer with a reasonable bounded utility function will pay, even while not paying in the standard Pascal’s mugging problem. (The naive model of the situation in which iterating the problem does not change how an expected utility maximizer handles it does not correctly model the interpretation of “iterated Pascal’s mugging” in which it makes sense to pay. I’d say what I mean, but actually keeping track of everything relevant to the problem makes it somewhat tedious to explain.)
It derives from the fact that median maximalisation doesn’t consider decisions independently, even if their gains and losses are independent.
For illustration, compare the following deal: you pay £q, and get £1 with probability p. There are n independent deals (assume your utility is linear in £).
If n=1, the median maximiser accepts the deal iff q0.5. Not a very good performance! Now let’s look at larger n. For m < n, accepting m deals gets you an expected reward of m(p-q). The median is a bit more complicated (see https://en.wikipedia.org/wiki/Binomial_distribution#Mode_and_median ), but it’s within £1 of the mean reward.
So if pq, it will accept all n deals.
For pq, it will accept at least n − 1/(p-q) deals. In all cases, its expected loss, compared with the mean maximiser, is less than £1.
There’s a similar effect going on when considering the seat-belt situation. Aggregation concentrates the distribution in a way that moved median and mean towards each other.
You appear to now be making an argument that you already conceded was incorrect in OP:
You then go on to say that if the agent also faces many decisions of a different nature, it won’t do that. That’s where I get lost.
The median maximiser accepts a 49.99999...% chance of death, only because “death”, “trivial cost” and “no cost” are the only options here. If I add “severe injury” and “light injury” to the outcomes, the maximiser will now accept less than a 49.9999...% chance of light injury. If we make light injury additive, and make the trivial cost also additive and not incomparable to light injuries, we get something closer to my illustrative example above.
Suppose it comes up with 2 possible policies, one of which involves a 49% chance of death and no chance of injury, and another which involves a 49% chance of light injury, and no chance of heavy injury or death. The median maximizer sees no reason to prefer the second policy if they have the same effects the other 51% of the time.
Er, yes, constructing single choice examples when the median behaves oddly/wrongly is trivial. My whole point is about what happens to median when you aggregate decisions.
You were claiming that in a situation where a median-maximizing agent has a large number of trivially inconvenient action that prevent small risks of death, heavy injury, or light injury, then it would accept a 49% chance of light injury, but you seemed to imply that it would not accept a 49% chance of death. I was trying to point out that this appears to be incorrect.
I’m not entirely sure what your objection is; we seem to be talking at cross purposes.
Let’s try it simpler. If we assume that the cost of buckling seat belts is incommensurable (in practice) with light injury (and heavy injury, and death), then the median maximising agent will accept a 49.99..% chance of (light injury or heavy injury or death), over their lifetime. Since light injury is much more likely than death, this in effect forces the probability of death down to a very low amount.
It’s just an illustration of the general point that median maximising seems to perform much better in real-world problems than its failure in simple theoretical ones would suggest.
No, it doesn’t. That does not address the fact that the agent will not preferentially accept light injury over death. Adopting a policy of immediately committing suicide once you’ve been injured enough to force you into the bottom half of outcomes does not decrease median utility. The agent has no incentive to prevent further damage once it is in the bottom half of outcomes. As a less extreme example, the value of house insurance to a median maximizer is 0, just because loosing your house is a bad outcome even if you get insurance money for it. This isn’t a weird hypothetical that relies on it being an isolated decision; it’s a real-life decision that a median maximizer would get wrong.
A more general way of stating how multiple decisions improve median maximalisation: the median maximaliser is indifferent of outcomes not at the median (eg suicide vs light injury). But as the decision tree grows and the number of possible situations does as well, the probability increases that outcomes not at the median in a one shot, will affect the median in the more complex situation.
This argument relies on your utility being a sum of effects from each of the decisions you made, but in reality, your decisions interact in much more complicated ways, so that isn’t a realistic model.
Also, if your defense of median maximization consists entirely of an argument that it approximates mean maximization, then what’s the point of all this? Why not just use expected utility maximization? I’m expecting you to bring up Pascal’s mugging here, but since VNM-rationality does not force you to pay the mugger, you’ll have to do better than that.
It doesn’t require that in the least. I don’t know if, eg, quadratic of higher order effects would improve or worsen the situation.
The consensus at the moment seems to be that if you have unbounded utility, it does force you to pay some muggers. Now, I’m perfectly fine with bounding your utility to avoid muggers, but that’s the kind of non-independent decision some people don’t like ;-)
The real problem is things like the Cauchy distribution, or any function without an expectation value at all. Saying “VNM works fine as long as we don’t face these difficult choices, then it breaks down” is very unsatisfactory. I’m also interested in seeing what happens when “expect to win” and “win in expectation” become quite distinct—a rare event, in practice.
The more concrete argument you made previous does rely on it. If what you’re saying now doesn’t, then I guess I don’t understand it.
I don’t follow. Maximizing the expected value of a bounded utility functions does respect independence.
That was an example. There’s another one in http://lesswrong.com/lw/1d5/expected_utility_without_the_independence_axiom/ which relies on “not risk loving”. That post doesn’t mention the median, but it does mention the standard deviation, and we know the mean must be within one SD of the mean (and often much closer).
Choosing to bound an unbounded utility function to avoid muggers does not.
That example also relies on your utility being the sum of components that are determined from your various actions.
To be clear, I was not suggesting that you have an unbounded utility function that it would make sense for you to maximize if it weren’t for Pascal’s mugger, so you should bound it when there might be a Pascal’s mugger around. I was suggesting that the utility function it makes sense for you to maximize is bounded. Unbounded utility functions are so loony they never should have been seriously considered in the first place; Pascal’s mugger is merely a dramatic illustration of that fact.
Edit: I probably shouldn’t rely on the theoretical reasons to prefer bounded utility functions, since they are not completely airtight and actual human preferences are more important anyway. So let’s look at actual human preferences. Suppose you’ve got a rational agent with preference relation “<”, and you want to test whether its utility function is bounded or unbounded. Here’s a simple test: First find outcomes A and B such that A<B (if you can’t even do that, its utility function is constant, hence bounded). Then pick an absurdly tiny probability p>0. Now see if you can find such a terrible C and such a wonderful D that pC+(1-p)B < pD + (1-p)A. If, for every p>0 you can find such C and D, then its utility function is unbounded. But if for some p>0, you cannot find any C and D that will suffice, even when you probe the extremes of goodness and badness, then its utility function is bounded. This test should sound familiar. What I’m getting at here is that one does not bound their unbounded utility function so that they don’t have to pay Pascal’s mugger; your preferences were simply bounded all along, and your response to Pascal’s mugger is proof.
Look, we’re arguing past each other here. My logical response here would be to add more options to the system, which would remove the problem you identified (and I don’t understand your house insurance example—this is just the seat-belt decision again as a one-shot, and I would address it by looking at all the financial decisions you make in your life—and if that’s not enough, all the decisions, including all the “don’t do something clearly stupid and pointless” ones).
What I think is clear is:
a) Median maximalisation makes bad decisions in isolated problems.
b) If we combine all the likely decisions that a median maximiser will have to make, the quality of the decisions increase.
If you want to argue against it, either say that a) is bad enough we should reject the approach anyway, even if it decides well in practice, or find examples where a real world median maximaliser will make bad decisions even in the real world (if you would pay Pascal’s mugger, then you could use that as an example).
We were modeling the seat-belt decision as something that makes the difference between being dead and being completely fine in the event of an accident (which I suppose is not very realistic, but whatever). I was trying to point to a situation where an event can happen which is bad enough to put in the bottom half of outcomes either way, so that nothing that happens conditional on the event can affect the median outcome, but a decision you can make ahead of time would make the difference between bad and worse.
I do think that a) is bad enough, because a decision procedure that does poorly in isolated problems is wrong, and thus cannot be expected to do well in realistic situations, as I mentioned previously. I guess b) is probably technically true, but it is not enough for the quality of the decisions to increase when the number increases; it should actually increase towards a limit that isn’t still awful, and come close to achieving that limit (I’m pretty sure it fails on at least one of those, though which step it fails on might depend on how you make things precise). I’ve given examples where median maximizers make bad decisions in the real world, but you’ve dismissed them with vague appeals to “everything will be fine when you consider it in the context of all the other decisions it has to make”.
And I’ve added in the specific other decisions needed to achieve this effect. I agree it’s not clear what exactly the median maximalisation converge on in the real world, but the examples you’ve produced are not sufficient to show it’s bad.
My take on this is that counterfactual decision count as well. ie if humans look not only at the decisions they face, but the ones they can imagine facing, then median maximalisation is improved. My justification for this line of thought is—how do you know that one chocolate cake is +10 utility while one coffee is +2 (and two coffees is +3, three is +2, and four is −1)? Not just the ordinal ranking, but the cardinality. I’d argue that you get this by either experiencing circumstances where you choose a 20% chance of a cake over coffee, or imagining yourself in that circumstance. And if imagination and past experiences are valid for the purpose of constructing your utility function, they should be valid for the purpose of median-maximalisation.
That you claim achieve that effect. But as I said, unless the are choices you can make that would protect you from light injury involve less inconvenience per % reduction in risk than the choices you can make that would protect you from death, it doesn’t work.
However, I did think of something which seems to sort of achieve what you want: if you have high uncertainty about what the value of your utility function will be, then adding something to it with some probability will have a significant effect on the median value, even if the probability is significantly less than 50%. For instance, a 49% chance of death is very bad because if there’s a 49% chance you die, then the median outcome is one in which you’re alive but in a worse situation than all but 1⁄51 of the scenarios in which you die. It may be that this is what you had in mind, and adding future decisions that involve uncertainty was merely a mechanism by which large uncertainty about the outcome was introduced, in which case future-you actually getting to make any choices about them was a red herring. I still don’t find this argument convincing either, though, both because it still undervalues protection from risks of losses that are large relative to the rest your uncertainty about the value of the outcome (for instance, note that when valuing reductions in risk of death, there is still a weird discontinuity around 50%), and because it assumes that you can’t make decisions that selectively have significant consequences only in very good or very bad outcomes (this is what I was getting at with the house insurance example).
I don’t understand what you’re saying here. Is it that you can maximize the median value of the mean of the values of your utility function in a bunch of hypothetical scenarios? If so, that sounds kind of like Houshalter’s median of means proposal, which approaches mean maximization as the number of samples considered approaches infinity.
The observation I have is that when facing many decisions, median maximialisation tends to move close to mean maximalisation (since the central limit theorem has “convergence in the distribution”, the median will converge to the mean in the case of averaging repeated independent processes; but there are many other examples of this). Therefore I’m considering what happens if you add “all the decisions you can imagine making” to the set of actual decisions you expect to make. This feels like it should move the two even closer together.
Ah, are you saying you should use your prior to choose a policy that maximizes your median utility, and then implementing that policy, rather than updating your prior with your observations and then choosing a policy that maximizes the median? So like UDT but with medians?
It seems difficult to analyze how it would actually behave, but it seems likely to be true that it acts much more similarly to mean utility maximization than it would if you updated before choosing the policy. Both of these properties (difficulty to analyze, and similarity to mean maximization) make it difficult to identify problems that it would perform poorly on. But this also makes it difficult to defend its alleged advantages (for instance, if it ends up being too similar to mean maximization, and if you use an unbounded utility function as you seem to insist, perhaps it pays Pascal’s mugger).
Ouch! Sorry for not being clear. If you missed that, then you can’t have understood much of what I was saying!
How do you know that it’s right to buckle your seatbelt? If you are only going to ride in a car once, never again. And there are no other risks to your life, and so no need to make a general policy against taking small risks?
I’m not confident that it’s actually the wrong choice. And if it is, it shouldn’t matter much. 99.99% of the time, the median will come out with higher utility than the EU maximizer.
This is generalizable. If there was a “utility competition” between different decision policies in the same situations, the median utility would usually come out on top. As the possible outcomes become more extreme and unlikely, expected utility will do worse and worse. With pascal’s mugging at the extreme.
That’s because EU trades away utility from the majority of possible outcomes, to really really unlikely outcomes. Outliers can really skew the mean of a distribution, and EU is just the mean.
Of course median can be exploited too. Perhaps there is some compromise between them that gets the behavior we want. There are an infinite number of possible policies for deciding which distribution of utilities to prefer.
EU was chosen because it is the only one that meets a certain set of conditions and is perfectly consistent. But if you allow for algorithms that select overall policies instead of decisions, like OP does, then you can make many different algorithms consistent.
So there is no inherent reason to prefer mean over median. It just comes down to personal preference, and subjective values. What probability distribution of utilities do you prefer?
I do think that the isolation of the decision is a red herring, but for the sake of the point I was trying to make, it is probably easier to replace the example with a structurally similar one in which the right answer is obvious: suppose you have the opportunity to press a button that will kill you will 49% probability, and give you $5 otherwise. This is the only decision you will ever make. Should you press the button?
As I was saying in my previous comment, I think that’s the wrong approach. It isn’t enough to kludge together a decision procedure that does what you want on the problems you thought of, because then it will do something you don’t want on something you haven’t thought of. You need a decision procedure that will reliably do the right thing, and in order to get that, you need it to do the right thing for the right reasons. EU maximization, applied properly, will tell you to do the correct things, and will do so for the correct reasons.
Actually, there is: https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem
Yes I said that median utility is not optimal. I’m proposing that there might be policies better than both EU or median.
Please reread the OP and my comment. If you allow selection over policies instead of individual decisions, you can be perfectly consistent. EU and median are both special cases of ways to pick policies, based on the probability distribution of utility they produce.
There is no law of the universe that some procedures are correct and others aren’t. You just have to pick one that you like, and your choice is going to be arbitrary.
If you go with EU you are pascal muggable. If you go with median you are muggable in certain cases as well (though you should usually, with >50% probability, end up with better outcomes in the long run. Whereas EU could possibly fail 100% of the time. So it’s exploitable, but it’s less exploitable at least.)
I don’t see how selecting policies instead of actions removes the motivation for independence.
Ultimately, it isn’t the policy that you care about; it’s the outcome. So you should pick a policy because you like the probability distributions over outcomes that you get from implementing it more than you like the probability distributions over outcomes that you would get from implementing other policies. Since there are many decision problems to use your policy on, this quite heavily constrains what policy you choose. In order to get a policy that reliably picks the actions that you decide are correct in the situations where you can tell what the correct action is, it will have to make those decisions for the same reason you decided that it was the best action (or at least something equivalent to or approximating the same reason). So no, the choice of policy is not at all arbitrary.
That is not true. EU maximizers with bounded utility functions reject Pascal’s wager.
There are two reasons to like independence. First of all, you might like it for philosophical/aesthetic reasons: “these things really should be independent, these really should be irrelevant”. Or you could like it because it prevents you from being money pumped.
When considering policies, money pumping is (almost) no longer an issue, because a policy that allows itself to be money-pumped is (almost) certainly inferior to one that doesn’t. So choosing policies removes one of the motivations for independence, to my mind the important one.
While it’s true that this does not tell you to pay each time to switch the outcomes around in a circle over and over again, it still falls prey to one step of a similar problem. Suppose their are 3 possible outcomes: A, B, and C, and there are 2 possible scenarios: X and Y. In scenario X, you get to choose between A and B. In scenario Y, you can attempt to choose between A and B, and you get what you picked with 50% probability, and you get outcome C otherwise. In each scenario, this is the only decision you will ever make. Suppose in scenario X, you prefer A over B, but in scenario Y, you prefer (B+C)/2 over (A+C)/2. But suppose you had to pay to pick A in scenario X, and you had to pay to pick (B+C)/2 in scenario Y, and you still make those choices. If Y is twice as likely as X a priori, then you are paying to get a probability distribution over outcomes that you could have gotten for free by picking B given X, and (A+C)/2 given Y. Since each scenario only involves you ever getting to make one decision, picking a policy is equivalent to picking a decision.
Your example is difficult to follow, but I think you are missing the point. If there is only one decision, then it’s actions can’t be inconsistent. By choosing a policy only once—one that maximizes it’s desired probability distribution of utility outcomes—it’s not money pumpable, and it’s not inconsistent.
Now by itself it still sucks because we probably don’t want to maximize for the best median future. But it opens up the door to more general policies for making decisions. You no longer have to use expected utility if you want to be consistent. You can choose a tradeoff between expected utility and median utility (see my top level comment), or a different algorithm entirely.
If there is only one decision point in each possible world, then it is impossible to demonstrate inconsistency within a world, but you can still be inconsistent between different possible worlds.
Edit: as V_V pointed out, the VNM framework was designed to handle isolated decisions. So if you think that considering an isolated decision rather than multiple decisions removes the motivation for the independence axiom, then you have misunderstood the motivation for the independence axiom.
I understand the two motivations for the independence axiom, and the practical one (“you can’t be money pumped”) is much more important that the theoretical one (“your system obeys this here philosophically neat understanding of irrelevant information”).
But this is kind of a moot point, because humans don’t have utility functions. And therefore we will have to construct them. And the process of constructing them is almost certainly going to depend on facts about the world, making the construction process almost certainly inconsistent between different possible worlds.
It shouldn’t. If your preferences among outcomes depend on what options are actually available to you, then I don’t see how you can justify claiming to have preferences among outcomes, as opposed to tendencies to make certain choices.
Then define me a process that takes people’s current mess of preferences, makes these into utility functions, and, respecting bounded rationality, is independent of options available in the real world. Even then, we have the problem that this mess of preferences is highly dependent on real world experiences in the first place.
If I always go left at a road, I have tendency to make certain choices. If I have a full model of the entire universe with labelled outcomes ranked on a utility function, and use it with unbounded rationality to make decisions, I have preferences among outcomes. The extremes are clear.
I feel that a bounded human being with a crude mental model that is trying to achieve some goal, imperfectly (because of ingrained bad habits, for instance) is better described as having preferences among outcomes. You could argue that they have mere tendencies, but this seems to stretch the term. But in any case, this is a simple linguistic dispute. Real human beings cannot achieve independence.
Define me a process with all those properties except the last one. If you can’t do that either, it’s not the last constraint that is to blame for the difficulty.
Yes, different agents have different preferences. The same agent shouldn’t have its preferences change when the available outcomes do.
If you are neutral between .4A+.6C and .4B+.6C, then you don’t have a very good claim to preferring A over B.
Well, there’s my old idea here: http://lesswrong.com/lw/8qb/cevinspired_models/ . I don’t think it’s particularly good, but it does construct a utility function, and might be doable with good enough models or a WBE. More broadly, there’s the general “figure out human preferences from their decisions and from hypothetical questions and fit a utility function to it”, which we can already do today (see “inverse reinforcement learning”); we just can’t do it well enough, yet, to get something generally safe at the other end.
None of these ideas have independent variants (not technically true; I can think of some independent versions of them, but they’re so ludicrously unsafe in our world that we’d rule them out immediately; thus, this would be a non-independent process).
?
If I actually do prefer A over B (and my behaviour reflects that in (1- ɛ)A+ ɛC versus (1-ɛ)B+ ɛC cases), then I have an extremely good claim to preferring A over B, and an extremely poor claim to independence.
I assumed accuracy was implied by “making a mess of preferences into a utility function”.
I’m somewhat skeptical of that strategy for learning utility functions, because the space of possible outcomes is extremely high-dimensional, and it may be difficult to test extreme outcomes because the humans you’re trying to construct a utility function for might not be able to understand them. But perhaps this objection doesn’t get to the heart of the matter, and I should put it aside for now.
I am admittedly not well-versed in inverse reinforcement learning, but this is a perplexing claim. Except for a few people like you suggesting alternatives, I’ve only ever heard “utility function” used to refer to a function you maximize the expected value of (if you’re trying to handle uncertainty), or a function you just maximize the value of (if you’re not trying to handle uncertainty). In the first case, we have independence. In the second case, the question of whether or not we obey independence doesn’t really make sense. So if inverse reinforcement learning violates independence, then what exactly does it try to fit to human preferences?
Then if the only difference between two gambles is that one might give you A when the other might give you B, you’ll take the one that might give you something you like instead of something you don’t like.
To be clear, I am saying the process of constructing the utility function violates independence, not that subsequently maximising it does. Similarly, choosing a median-maximising policy P violates independence, but there is (almost certainly) a utility u such that maximising u is the same as following P.
Once the first choice is made, we have independence in both cases; before it is made, we have it in neither. The philosophical underpinning of independence in single decisions therefore seems very weak.
Feel free to tell me to shut up and learn how inverse reinforcement learning works before bothering you with such questions, if that is appropriate, but I’m not sure what you mean. Can you be more precise about what property you’re saying inverse reinforcement learning doesn’t have?
Inverse reinforcement learning relies on observation of humans performing specific actions, and drawing the “right” conclusion as to what their preferences. Indirectly, it relies on humans tinkering with its code to remove “errors”, ie things that don’t fit with the mental image that human programmers of what preferences should be.
Given that human desires are not independent (citation not needed), this process, if it produces a utility function, involves constructing something independent from non-independent input. However, to establish this utility function, the algorithm has access only to the particular problems given to it, and the particular mental images of its programmers. It is almost certain that the end result would be somewhat different if it was trained on different problems, or if its programmers had different intuitions. Therefore the process itself cannot be independent.
Ah, I see what you mean, and you’re right; the utility function constructed will depend on how the data points are sampled. This isn’t quite the same as saying that the result will depend on what results are actually available, though, unless knowledge about what results will be available is used to determine how to sample the data. This still seems like somewhat of a defect of inverse reinforcement learning, unless there ends up being a good case that some particular way of sampling the data is optimal for revealing underlying preferences and ignoring biases, or something like that.
That’s probably true, but on the other hand, you seem to want to pin the deviations of human behavior from VNM rationality on violations of the independence axiom, and it isn’t clear to me that this is the case (I don’t think the point you were making relies on this, so if you weren’t trying to make that claim then you can ignore this; it just seemed like you might be). There are situations where there are large framing effects (that is, whether A or B is preferred depends on how the options are presented, even if no other outcome C is being mixed in with them), and likely also violations of transitivity (where someone would say A>B, B>C, and C>A whenever you ask them about 2 of them without bringing up the third). It seems likely to me that most paradoxes of human decision-making have more to do with these than they do to violations of independence.
It can’t be inconsistent within a world no matter how many decisions points there are. If we agree it’s not inconsistent, then what are you arguing against?
I don’t care about the VNM framework. As you said, it is designed to be optimal for decisions made in isolation. Because we don’t need to make decisions in isolation, we don’t need to be constrained by it.
No. Inconsistency between different possible worlds is still inconsistency.
The difference doesn’t matter that much in practice. If there are multiple decision points, you can combine them into one by selecting a policy, or by considering them sequentially and using your beliefs about what your choices will be in the future to compute the expected utilities of the possible decisions available to you now. The reason that the VNM framework was designed for one-shot decisions is that it makes things simpler without actually constraining what it can be applied to.
It’s perfectly consistent in the sense that it’s not money pumpable, and always makes the same decisions given the same information. It will make different decisions in different situations, given different information. But that is not inconsistent by an reasonable definition of “inconsistent”.
It makes a huge difference. If you want to get the best median future, then you can’t make decisions in isolation. You need to consider every possible decision you will have to make, and their probability. And choose a decision policy that selects the best median outcome.
As in my previous example (sorry about it being difficult to follow, though I’m not sure yet what I could say to clarify things), it is inconsistent in the sense that it can lead you to pay for probability distributions over outcomes that you could have achieved for free.
Right. As I just said, “you can… consider them sequentially and use your beliefs about what your choices will be in the future to compute the expected utilities of the possible decisions available to you now.” (edited to fix grammar). This reduces iterated decisions to isolated decisions: you have certain beliefs about what you’ll do in the future, and now you just have to make a decision on the issue facing you now.