This might sound silly, but it’s deeper than it looks: the reason why we use the expected value of utility (i.e. means) to determine the best of a set of gambles is because utility is defined as the thing that you maximize the expected value of.

The thing that’s nice about VNM utility is that it’s mathematically consistent. That means we can’t come up with a scenario where VNM utility generates silly outputs with sensible inputs. Of course we can give VNM silly inputs and get silly outputs back—scenarios like Pascal’s Mugging are the equivalent of “suppose something really weird happens; wouldn’t that be weird?” to which the answer is “well, yes.”

The really nice thing about VNM is that it’s the only rule that’s mathematically consistent with itself and a handful of nice axioms. You might give up one of those axioms, but for any of those axioms we can show an example where a rule that doesn’t follow that axiom will take sensible inputs and give silly outputs. So I don’t think there’s much to be gained by trying to replace a mean decision rule with a median decision rule or some other decision rule—but there is a lot to be gained by sharpening our probability distributions and more clearly figuring out our mapping from world-histories to utilities.

“suppose something really weird happens; wouldn’t that be weird?”

To me, consequentialism is either something trivial or something I reject, but that said this is a fully general (and kind of weak) counter-argument. I can apply it to Newcomb from the CDT point of view. I can apply it to Smoking Lesion from the EDT point of view! I can apply it to astronomic data from the Ptolemy’s theory of celestial motion point of view! We have to deal with things!

To me, consequentialism is either something trivial or something I reject

I hesitate to call consequentialism trivial, because I wouldn’t use it to describe a broad class of ‘intelligent’ agents, but I also wouldn’t reject it, because it does describe the design of those agents.

this is a fully general (and kind of weak) counter-argument.

I don’t see it as a counter-argument. In general, I think that a method is appropriate if hard things are hard and easy things are easy—and, similarly, normal things are normal and weird things are weird. If the output is weird in the same way that the input is weird, the system is behaving appropriately; if it adds or subtracts weirdness, then we’re in trouble!

For example, suppose you supplied a problem with a relevant logical contradiction to your decision algorithm, and it spat out a single numerical answer. Is that a sign of robustness, or lack of robustness?

I just meant I accept the consequentialist idea in decision theory that we should maximize, e.g. pick the best out of alternatives. But said in this way, it’s a trivial point. I reject more general varieties of consequentialism (for reasons that are not important right now, but basically I think a lot of weird conclusions of consequentialism are due to modeling problems, e.g. the set up that makes consequentialism work doesn’t apply well).

normal things are normal and weird things are weird

I don’t know what you are saying here. Can you taboo “weird?” Newcomb is weird for CDT because it explicitly violates an assumption CDT is using. The answer here is to go meta and think about a family of decision theories of which CDT is one, indexed by their assumption sets.

I just meant I accept the consequentialist idea in decision theory that we should maximize, e.g. pick the best out of alternatives. But said in this way, it’s a trivial point.

I understood and agree with that statement of consequentialism in decision theory—what I disagree with is that it’s trivial that maximization is the right approach to take! For many situations, a reflexive agent that does not actively simulate the future or consider alternatives performs better than a contemplative agent that does simulate the future and considerate alternatives, because the best alternative is “obvious” and the acts of simulation and consideration consume time and resources that do not pay for themselves.

That’s obviously what’s going on with thermostats, but I would argue is what goes on all the way up to the consequentialism-deontology divide in ethics.

Can you taboo “weird?”

I would probably replace it with Pearl’s phrase here, of “surprising or unbelievable.”

To use the specific example of Newcomb’s problem, if people find a perfect predictor “surprising or unbelievable,” then they probably also think that the right thing to do around a perfect predictor is “surprising or unbelievable,” because using logic on an unbelievable premise can lead to an unbelievable conclusion! Consider a Mundane Newcomb’s problem which is missing perfect prediction but has the same evidential and counterfactual features: that is, Omega offers you the choice of one or two boxes, you choose which boxes to take, and then it puts a million dollars in the red box and a thousand dollars in the blue box if you choose only the red box and it puts a thousand dollars in the blue box if you choose the blue box or no boxes. Anyone that understands the scenario and prefers more money to less money will choose just the red box, and there’s nothing surprising or unbelievable about it.

What is surprising is the claim that there’s an entity who can replicate the counterfactual structure of the Mundane Newcomb scenario without also replicating the temporal structure of that scenario. But that’s a claim about physics, not decision theory!

because the best alternative is “obvious” and the acts of simulation and consideration consume time and
resources that do not pay for themselves.

Absolutely. This is the “bounded rationality” setting lots of people think about. For instance, Big Data is fashionable these days, and lots of people think about how we may do usual statistics business under severe computational constraints due to huge dataset sizes, e.g. stuff like this:

But in bounded rationality settings we still want to pick the best out of our alternatives, we just have a constraint that we can’t take more than a certain amount of resources to return an answer. The (trivial) idea of doing your best is still there. That is the part I accept. But that part is boring, thinking of the right thing to maximize is what is very subtle (and may involve non-consequentialist ideas, for example a decision theory that handles blackmail may involve virtue ethical ideas because the returned answer depends on “the sort of agent” someone is).

I don’t agree. Utility is a separate concept from expected value maximization. Utility is a way of ordering and comparing different outcomes based on how desirable they are. You can say that one outcome is more desirable than another, or even quantify how many times more desirable it is. This is a useful and general concept.

Expected utility does have some nice properties being completely consistent. However I argued above that this isn’t a necessary property. It adds complexity, sure, but if you self modify your decision making algorithm or predetermine your actions, you can force your future self to be consistent with your present self’s desires.

Expected utility is perfectly rational as the number of “bets” you take goes to infinity. Rewards will cancel out the losses in the limit, and so any agent would choose to follow EU regardless of their decision making algorithm. But as the number of bets becomes finite, it’s less obvious that this is the most desirable strategy.

That means we can’t come up with a scenario where VNM utility generates silly outputs with sensible inputs. Of course we can give VNM silly inputs and get silly outputs back—scenarios like Pascal’s Mugging are the equivalent of “suppose something really weird happens; wouldn’t that be weird?” to which the answer is “well, yes.”

Pascal’s Mugging isn’t “weird”, it’s perfectly typical. There are probably an infinite number of pascal’s mugging type situations. Hypotheses with exceedingly low probability but high utility.

If we built an AI today, based on pure expected utility, it would most likely fail spectacularly. These low probability hypotheses would come to totally dominate it’s decisions. Perhaps it would start to worship various gods and practice rituals and obeying superstitions. Or something far more absurd we haven’t even thought of.

And if you really believe in EU, you can’t say that this behavior is wrong or undesirable. This is what you should be doing, if you could, and you are losing a huge amount of EU by not doing it. You should want more than anything in existence, the ability to exactly calculate these hypotheses so you can collect that EU.

I don’t want that though. I want a decision rule such that I am very likely to end up in a good outcome. Not one where I will mostly likely end up in a very suboptimal outcome, with an infinitesimal probability of winning the infinite utility lottery.

Expected utility is convenient and makes for a nice mathematical theory.

It also makes a lot of assumptions. One assumes that the expectation does, in fact, exist. It need not. For example, in a game where two players toss a fair coin, we expect that in the long run the number of heads should equal the number of tails at some point. It turns out that the expected waiting time is infinite. Then there’s the classic St. Petersburg paradox.

There are examples of “fair” bets (i.e. expected gain is 0) that are nevertheless unfavorable (in the sense that you’re almost certain to sustain a net loss over time).

Expected utility is a model of reality that does a good job in many circumstances but has some key drawbacks where naive application will lead to unrealistic decisions. The map is not the territory, after all.

Utility is a separate concept from expected value maximization. Utility is a way of ordering and comparing different outcomes based on how desirable they are.

To Bentham, sure; today, we call something that generic “ranking” or something similar, because VNM-utility is the only game in town when it comes to assigning real-valued desirabilities to consequences.

But as the number of bets becomes finite, it’s less obvious that this is the most desirable strategy.

Disagreed. The proof of the VNM axioms goes through for a single bet; I recommend you look that up, and then try to create a counterexample.

Note that it’s easy to come up with a wrong utility mapping. One could, say, map dollars linearly to utility and then say “but I don’t prefer a half chance of $100 and half chance of nothing to a certain $50!” , but that’s solved by changing the utility mapping from linear from sublinear (say, log or sqrt or so on). In order to exhibit a counterexample it has to look like the Allais paradox, where someone confirms two preferences and then does not agree with the consequence of those preferences considered together.

There are probably an infinite number of pascal’s mugging type situations. Hypotheses with exceedingly low probability but high utility.

It probably isn’t the case that there are an infinite number of situations where the utility times the probability is higher than the cost, and if there are, that’s probably a faulty utility function or faulty probability estimator rather than a faulty EU calculation. Consider this bit from You and Your Research by Hamming:

Let me warn you, `important problem’ must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important. When I say that most scientists don’t work on important problems, I mean it in that sense.

An AI might correctly calculate that time travel is the most positive technology it could possibly develop—but also quickly calculate that it has no idea where to even start, and so the probability of success from thinking about it more is low enough that it should go for a more credible option. That’s what human thinkers do and it doesn’t seem like a mistake in the way that the Allais paradox seems like a mistake.

But as the number of bets becomes finite, it’s less obvious that this is the most desirable strategy.

Disagreed. The proof of the VNM axioms goes through for a single bet; I recommend you look that up, and then try to create a counterexample.

Pascal’s wager is the counterexample, and it’s older than VNM. EY’s Pascal’s mugging was just an attempt to formalize it a bit more and prevent silly excuses like “well what if we don’t allow infinites or assume the probabilities exactly cancel out.”

Counterexample in that it violates what humans want, not that it produces inconsistent behavior or anything. It’s perfectly valid for an agent to follow EU, as it is for it to follow my method. What we are arguing about is entirely subjective.

If you really believe in EU a priori, then no argument should be able to convince you it is wrong. You would find nothing wrong with Pascal situations, and totally agree with the result of EU. You wouldn’t have to make clever arguments about the utility function or probability estimates to get out of it.

It probably isn’t the case that there are an infinite number of situations where the utility times the probability is higher than the cost, and if there are, that’s probably a faulty utility function or faulty probability estimator rather than a faulty EU calculation.

This is pretty thoroughly argued in the original Pascal’s Mugging post. Hypotheses of vast utility can grow much faster than their improbability. The hypothesis “you will be rewarded/tortured 3^^^3 units” is infinitesimally smaller in an EU calculation to the hypothesis “you will be rewarded/tortured 3^^^^^^^3 units”, and only takes a few more bits to express, and it can grow even further.

Pascal’s wager is the counterexample, and it’s older than VNM.

Counterexample in what sense? If you do in fact receive infinite utility from going to heaven, and being Christian raises the chance of you going to heaven by any positive amount over your baseline chance, then it is the right move to be Christian instead of baseline.

The reason people reject Pascal’s Wager or Mugging is, as I understand it, they don’t see the statement “you receive infinite utility from X” or “you receive a huge amount of disutility from Y” as actual evidence about their future utility.

In general, I think that any problem which includes the word “infinite” is guilty until proven innocent, and it is much better to express it as a limit. (This clears up a huge amount of confusion.) And the general principle- that as the prize for winning a lottery gets better, the probability of winning the lottery necessary to justify buying a fixed-price ticket goes down, seems like a reasonable principle to me.

It’s perfectly valid for an agent to follow EU, as it is for it to follow my method. What we are arguing about is entirely subjective.

I think money pumps argue against subjectivity. Basically, if you use an inconsistent decision theory, someone else can make money off your inconsistency or you don’t actually use that inconsistent decision theory.

If you really believe in EU a priori, then no argument should be able to convince you it is wrong. You would find nothing wrong with Pascal situations, and totally agree with the result of EU. You wouldn’t have to make clever arguments about the utility function or probability estimates to get out of it.

I will say right now: I believe that if you have a complete set of outcomes with known utilities and the probabilities of achieving those outcomes conditioned on taking actions from a set of possible actions, the best action in that set is the one with the highest probability-weighted utility sum. That is, EU maximization works if you feed it the right inputs.

Do I think it’s trivial to get the right inputs for EU maximization? No! I’m not even sure it’s possible except in approximation. Any problem that starts with utilities in the problem description has hidden the hard work under the rug, and perhaps that means they’ve hidden a ridiculous premise.

Hypotheses of vast utility can grow much faster than their improbability.

Assuming a particular method of assigning prior probabilities to statements, yes. But is that the right method of assigning prior probabilities to statements?

(That is, yes, I’ve read Eliezer’s post, and he’s asking how to generate probabilities of consequences given actions. That’s a physics question, not a decision theory question.)

If you do in fact receive infinite utility from going to heaven, and being Christian raises the chance of you going to heaven by any positive amount over your baseline chance, then it is the right move to be Christian instead of baseline.

Where “right” is defined as “maximizing expected utility”, then yes. It’s just a tautology, “maximizing expected utility maximizes expected utility”.

My point is if you actually asked the average person, even if you explained all this to them, they would still not agree that it was the right decision.

There is no law written into the universe that says you have to maximize expected utility. I don’t think that’ what humans really want. If we choose to follow it, in many situation it will lead to undesirable outcomes. And it’s quite possible that those situations are actually common.

It may mean life becomes more complicated than making simple EU calculations, but you can still be perfectly consistent (see further down.)

In general, I think that any problem which includes the word “infinite” is guilty until proven innocent, and it is much better to express it as a limit. (This clears up a huge amount of confusion.)

You could express it as a limit trivially (e.g. a hypothesis that in heaven you will collect 3^^^3 utilons per second for an unending amount of time.)

And the general principle- that as the prize for winning a lottery gets better, the probability of winning the lottery necessary to justify buying a fixed-price ticket goes down, seems like a reasonable principle to me.

Sounds reasonable, but it breaks down in extreme cases, where you end up spending almost all of your probability mass in exchange for a single good future with arbitrarily low probability.

Here’s a thought experiment. Omega offers you tickets for 2 extra lifetimes of life, in exchange for a 1% chance of dying when you buy the ticket. You are forced to just keep buying tickets until you finally die.

Maybe you object that you discount extra years of life by some function, so just modify the thought experiments so the reward increase factorially per ticket bought, or something like that.

Fortunately we don’t have to deal with these situations much, because we happen to live in a universe where there aren’t powerful agents offering us very high utility lotteries. But these situations occur all the time if you deal with hypotheses instead of lotteries. The only reason we don’t notice it is because we ignore or refuse to assign probability estimates to very unlikely hypotheses. An AI might not, and so it’s very important to consider this issue.

I think money pumps argue against subjectivity. Basically, if you use an inconsistent decision theory, someone else can make money off your inconsistency or you don’t actually use that inconsistent decision theory.

My method isn’t vulnerable to money pumps, as is an infinite number of arbitrary algorithms of the same class. See my comment here for details.

You don’t even need the stuff I wrote about predetermining actions, that just minimizes regret. Even a naive implementation of expected median utility should not be money pumpable.

Assuming a particular method of assigning prior probabilities to statements, yes. But is that the right method of assigning prior probabilities to statements?

The method by which you assign probabilities should be unrelated to the method you assign utilities to outcomes. That is, you can’t just say you don’t like the outcome EU gives you and so assign it a lower probability, that’s a horrible violation of Bayesian principles.

I don’t know what the correct method of assigning probabilities, but even if you discount complex hypotheses factorially or something, you still get the same problem.

I certainly think these scenarios have reasonable prior probability. God could exist, we could be in the matrix, etc. I give them so low probability I don’t typically think about them, but for this issue that is irrelevant.

It’s just a tautology, “maximizing expected utility maximizes expected utility”.

Yes. That’s the thing that sounds silly but is actually deep.

Here’s a thought experiment. Omega offers you tickets for 2 extra lifetimes of life, in exchange for a 1% chance of dying when you buy the ticket. You are forced to just keep buying tickets until you finally die.

Maybe you object that you discount extra years of life by some function, so just modify the thought experiments so the reward increase factorially per ticket bought, or something like that.

That is the objection, but I think I should explain it in a more fundamental way.

What is the utility of a consequence? For simplicity, we often express it as a real number, with the caveat that all utilities involved in a problem have their relationships preserved by an affine transformation. But that number is grounded by a gamble. Specifically, consider three consequences, A, B, and C, with u(A)<u(B)<u(C). If I am indifferent between B for certain and A with probability p and C otherwise, I encode that with the mathematical relationship:

u(B)=p u(A)+(1-p) u(C)

As I express more and more preferences, each number is grounded by more and more constraints.

The place where counterexamples to EU calculations go off the rails is when people intervene at the intermediate step. Suppose p is 50%, and I’ve assigned 0 to A, 1 to B, and 2 to C. If a new consequence, D, is introduced with a utility of 4, that immediately implies:

I am indifferent between (50% A, 50% D) and (100% C).

I am indifferent between (75% A, 25% D) and (100% B).

I am indifferent between (67% B, 33% D) and (100% C).

If one of those three statements is not true, I can use that and D having a utility of 4 to prove a contradiction. But the while the existence of D and my willingness to accept those specific gambles implies that D’s utility is 4, the existence of the number 4 does not imply that there exists a consequence where I’m indifferent to those gambles!

And so very quickly Omega might have to offer me a lifetime longer than the lifetime of the universe, and because I don’t believe that’s possible I say “no thanks, I don’t think you can deliver, and in the odd case where you can deliver, I’m not sure that I want what you can deliver.” (This is the resolution of the St. Petersburg Paradox where you enforce that the house cannot pay you more than the total wealth of the Earth, in which case the expected value of the bet comes out to a reasonable, low number of dollars, roughly where people estimate the value of the bet.)

But these situations occur all the time if you deal with hypotheses instead of lotteries. The only reason we don’t notice it is because we ignore or refuse to assign probability estimates to very unlikely hypotheses. An AI might not, and so it’s very important to consider this issue.

To me, this maps on to basic research. There’s some low probability that a particular molecule cures a particular variety of cancer, but it would be great if it did—so let’s check. The important conceptual addition from this analogy is that both hypotheses are entangled (this molecule curing cancer implies things about other molecules) and values are entangled (the second drug that can cure a particular variety of cancer is less valuable than the first drug that can).

And so an AI might have somewhat different basic research priorities than we do—indeed, humans vary widely on their preferences for following various uncertain paths—but it seems likely to me that it could behave reasonably when coming up with a portfolio of actions to take, even if it looks like it might behave oddly with only one option.

The method by which you assign probabilities should be unrelated to the method you assign utilities to outcomes. That is, you can’t just say you don’t like the outcome EU gives you and so assign it a lower probability, that’s a horrible violation of Bayesian principles.

Playing against nature, sure—but playing against an intelligent agent? Running a minimax calculation to figure out that my opponent is not likely to let me checkmate him in one move is hardly a horrible violation of Bayesian principles!

And so very quickly Omega might have to offer me a lifetime longer than the lifetime of the universe, and because I don’t believe that’s possible I say “no thanks, I don’t think you can deliver, and in the odd case where you can deliver, I’m not sure that I want what you can deliver.”

the house cannot pay you more than the total wealth of the Earth, in which case the expected value of the bet comes out to a reasonable, low number of dollars, roughly where people estimate the value of the bet.

This is a cop out. Obviously that specific situation can’t occur in reality, that’s not the point. If your decision algorithm fails in some extreme cases, at least confess that it’s not universal.

And so very quickly Omega might have to offer me a lifetime longer than the lifetime of the universe, and because I don’t believe that’s possible I say “no thanks, I don’t think you can deliver

Same thing. Omega’s ability and honesty are premises.

The point of the thought experiment is just to show that EU is required to trade away huge amounts of outcome-space for really good but improbable outcomes. This is a good strategy if you plan on making an infinite number of bets, but horrible if you don’t expect to live forever.

I don’t get your drug research analogy. There is no pascal’s equivalent situation in drug research. At best you find a molecule that cures all diseases, but that’s hardly infinite utility.

Instead it would be more like, “there is a tiny, tiny probability that a virus could emerge which causes humans not to die, but to suffer for eternity in the worst pain possible. Therefore, by EU calculation, I should spend all of my resources searching for a possible vaccine for this specific disease, and nothing else.”

This is a cop out. Obviously that specific situation can’t occur in reality, that’s not the point. If your decision algorithm fails in some extreme cases, at least confess that it’s not universal.

What does it mean for a decision algorithm to fail? I’ll give an answer later, but here I’ll point out that I do endorse that multiplication of reals is universal—that is, I don’t think multiplication breaks down when the numbers get extreme enough.

Same thing. Omega’s ability and honesty are premises.

And an unbelievable premise leads to an unbelievable conclusion. Don’t say that logic has broken down because someone gave the syllogism “All men are immortal, Socrates is a man, therefore Socrates is still alive.”

The point of the thought experiment is just to show that EU is required to trade away huge amounts of outcome-space for really good but improbable outcomes. This is a good strategy if you plan on making an infinite number of bets, but horrible if you don’t expect to live forever.

How does [logic work]? Eliezer puts it better than I can:

The power of logic is that it relates models and statements. … And here is the power of logic: For each syntactic step we do on our statements, we preserve the match to any model. In any model where our old collection of statements was true, the new statement will also be true. We don’t have to check all possible conforming models to see if the new statement is true in all of them. We can trust certain syntactic steps in general—not to produce truth, but to preserve truth.

EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to produce preferences but to preserve them. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.

And if it’s not true that you would trade away huge amounts of outcome-space for really good but improbable outcomes, this is a fact about your preference model that EU preserves! Remember, EU preference models map lists of outcomes to classes of lists of real numbers, but the inverse mapping is not guaranteed to have support over the entire reals.

I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm, and the chief ways to do so are 1) to do the math wrong and be inconsistent and 2) to make it more costly to express your preferences or world-model.

I don’t get your drug research analogy.

We have lots of hypotheses about low-probability, high-payout options, and if humans make mistakes, it is probably by overestimating the probability of low-probability events and overestimating how much we’ll enjoy the high payouts, both of which make us more likely to pursue those paths than a rational version of ourselves.

So it seems to me that if we have an algorithm that can correctly manage the budget of a pharmaceutical corporation, balancing R&D and marketing and production and so on, that requires solving this philosophical problem. But we have the mathematical tools to correctly manage the budget of a pharmaceutical corporation given a world-model, which says to me we should turn our attention to getting more precise and powerful world-models.

What does it mean for a decision algorithm to fail?

When it makes decisions that are undesirable. There is no point deciding to run a decision algorithm which is perfectly consistent but results in outcomes you don’t want.

In the case of the Omega’s-life-tickets scenario, one could argue it fails in an objective sense since it will never stop buying tickets until it dies. But that wasn’t even the point I was trying to make.

And an unbelievable premise leads to an unbelievable conclusion.

I don’t know if there is a name for this fallacy but there should be. Where someone objects to the premises of a hypothetical situation intended just to demonstrate a point. E.g. people who refuse to answer the trolley dilemma and instead say “but that will probably never happen!” It’s very frustrating.

EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to produce preferences but to preserve them. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.

This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.

What I’ve been saying from the very beginning is that there isn’t any reason to believe there is any utility function that will produce desirable outcomes if fed to an expected utility maximizer.

I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm

Even if you are an EU maximizer, EU will make you “predictably” worse off, as in the majority of cases you will be worse off. A true EU maximizer doesn’t care so long as the utility of the very low probability outcomes is high enough.

There are good and bad reasons to fight the hypothetical. When it comes to these particular problems, though, the objections I’ve given are my true objections. The reason I’d only pay a tiny amount of money for the gamble in the St. Petersburg Paradox is that there is only so much financial value that the house can give up. One of the reasons I’m sure this is my true objection is because the richer the house, the more I would pay for such a gamble. (Because there are no infinitely rich houses, there is no one I would pay an infinite amount to for such a gamble.)

This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.

I’m not sure why you think it’s subtle—I started off this conversation with:

This might sound silly, but it’s deeper than it looks: the reason why we use the expected value of utility (i.e. means) to determine the best of a set of gambles is because utility is defined as the thing that you maximize the expected value of.

But I don’t think it’s quite right to call it “circular,” for roughly the same reasons I don’t think it’s right to call logic “circular.”

What I’ve been saying from the very beginning is that there isn’t any reason to believe there is any utility function that will produce desirable outcomes if fed to an expected utility maximizer.

To make sure we’re talking about the same thing, I think an expected utility maximizer (EUM) is something that takes both a function u(O) that maps outcomes to utilities, a function p(A->O) that maps actions to probabilities of outcomes, and a set of possible actions, and then finds the action out of all possible A that has the maximum weighted sum of u(O)p(A->O) over all possible O.

So far, you have not been arguing that every possible EUM leads to pathological outcomes; you have been exhibiting particular combinations of u(O) and p(A->O) that lead to pathological outcomes, and I have been responding with “have you tried not using those u(O)s and p(A->O)s?”.

It doesn’t seem to me that this conversation is producing value for either of us, which suggests that we should either restart the conversation, take it to PMs, or drop it.

Here’s a thought experiment. Omega offers you tickets for 2 extra lifetimes of life, in exchange for a 1% chance of dying when you buy the ticket. You are forced to just keep buying tickets until you finally die.

This suggests buying tickets takes finite time per ticket, and that the offer is perpetually open. It seems like you could get a solid win out of this by living your life, buying one ticket every time you start running out of life. You keep as much of your probability mass alive as possible for as long as possible, and your probability of being alive at any given time after the end of the first “lifetime” is greater than it would’ve been if you hadn’t bought tickets. Yeah, Omega has to follow you around while you go about your business, but that’s no more obnoxious than saying you have to stand next to Omega wasting decades on mashing the ticket-buying button.

Expected utility is perfectly rational as the number of “bets” you take goes to infinity.

That’s not the way in which maximizing expected utility is perfectly rational.

The way it’s perfectly rational is this. Suppose you have any decision making algorithm; if you like, it can have an internal variable called “utility” that lets it order and compare different outcomes based on how desirable they are. Then either:

the algorithm has some ugly behavior with respect to a finite collection of bets (for instance, there are three bets A, B, and C such that it prefers A to B, B to C, and C to A), or

the algorithm is equivalent to one which maximizes the expected value of some utility function: maybe the one that your internal variable was measuring, maybe not.

The first condition is not true, since it gives a consistent value to any probability distribution of utilities. The second condition is not true other since the median function is not merely a transform of the mean function.

I’m not sure what the “ugly” behavior you describe is, and I bet it rests on some assumption that’s too strong. I already mentioned how inconsistent behavior can be fixed by allowing it to predetermine it’s actions.

You can find the Von Neumann—Morgenstern axioms for yourself. It’s hard to say whether or not they’re too strong.

The problem with “allowing [the median algorithm] to predetermine its actions” is that in this case, I no longer know what the algorithm outputs in any given case. Maybe we can resolve this by considering a case when the median algorithm fails, and you can explain what your modification does to fix it. Here’s an example.

Suppose I roll a single die.

Bet A loses you $5 on a roll of 1 or 2, but wins you $1 on a roll of 3, 4, 5, or 6.

Bet B loses you $5 on a roll of 5 or 6, but wins you $1 on a roll of 1, 2, 3, or 4.

Bet A has median utility of U($1), as does bet B. However, combined they have a median utility of U(-$4).

So the straightforward median algorithm pays money to buy Bet A, pays money to buy Bet B, but will then pay money to be rid of their combination.

I think I’ve found the core of our disagreement. I want an algorithm that considers all possible paths through time. It decides on a set of actions, not just for the current time step, but for all possible future time steps. It chooses such that the final probability distribution of possible outcomes, at some point in the future, is optimal according to some metric. I originally thought of median, but it can work with any arbitrary metric.

This is a generalization of expected utility. The VNM axioms require an algorithm to make decisions independently and Just In Time. Whereas this method lets it consider all possible outcomes. It may be less elegant than EU, but I think it’s closer to what humans actually want.

Anyway your example is wrong, even without predetermined actions. The algorithm would buy bet A, but then not buy bet B. This is because it doesn’t consider bets in isolation like EU, but considers it’s entire probability distribution of possible outcomes. Buying bet B would decrease it’s expected median utility, so it wouldn’t take it.

Assuming the bet has a fixed utility, then EU gives it a fixed estimate right away. Whereas my method considers it along with all other bets that it’s made or expects to make, and it’s estimate can change over time. I should have said that it’s not independent or fixed, but that is what I meant.

In the VNM scheme where expected utility is derived at a consequence of the axioms, the way that a bet’s utility changes over time is that its utility is not fixed. Nothing at all stops you from changing the utility you attach to a 50:50 gamble of getting a kitten versus $5 if your utility for a kitten (or for $5) changes: for example, if you get another kitten or win the lottery.

Generalizing to allow the value of the bet to change when the value of the options did not change seems strange to me.

This is a generalization of expected utility.

I am lost, this is just EU in a longitudinal setting? You can average over lots of stuff. Maximizing EU is boring, it’s specifying the right distribution that’s tricky.

It’s not EU, since it can implement arbitrary algorithms to specify the desired probability distribution of outcomes. Averaging utility is only one possibility, another I mentioned was median utility.

So you would take the median utility of all the possible outcomes. And then select the action (or series of actions in this case) that leads to the highest median utility.

No method of specifying utilities would let EU do the same thing, but you can trivially implement EU in it, so it’s strictly more general than EU.

I think I’ve found the core of our disagreement. I want an algorithm that considers all possible paths through time. It decides on a set of actions, not just for the current time step, but for all possible future time steps.

So, I think you might be interested in UDT. (I’m not sure what the current best reference for that is.) I think that this requires actual omniscience, and so is not a good place to look for decision algorithms.

(Though I should add that typically utilities are defined over world-histories, and so any decision algorithm typically identifies classes of ‘equivalent’ actions, i.e. acknowledges that this is a thing that needs to be accepted somehow.)

UDT is overkill. The idea that all future choices can be collapsed into a single choice appears in the work of von Neumann and Morgenstern, but is probably much older.

Oh, I see. I didn’t take that problem into account, because it doesn’t matter for expected utility, which is additive. But you’re right that considering the entire probability distribution is the right thing to do, and under than assumption we’re forced to be transitive.

The actual VNM axiom violated by median utility is independence: If you prefer X to Y, then a gamble of X vs Z is preferable to the equivalent gamble of Y vs Z. Consider the following two comparisons:

Taking bet A, as above, versus the status quo.

A ^{2}⁄_{3} chance of taking bet A and a ^{1}⁄_{3} chance of losing $5, versus a ^{2}⁄_{3} chance of the status quo and a ^{1}⁄_{3} chance of losing $5.

In the first case, bet A has median utility U($1) and the status quo has U($0), so you pick bet A. In the second case, a gamble with a possibility of bet A has median utility U(-$5) and a gamble with a possibility of the status quo still has U($0), so you pick the second gamble.

Of course, independence is probably the shakiest of the VNM axioms, and it wouldn’t surprise me if you’re unconvinced by it.

This might sound silly, but it’s deeper than it looks: the reason why we use the expected value of utility (i.e. means) to determine the best of a set of gambles is because utility is defined as the thing that you maximize the expected value of.

The thing that’s nice about VNM utility is that it’s mathematically consistent. That means we can’t come up with a scenario where VNM utility generates silly outputs with sensible inputs. Of course we can give VNM silly inputs and get silly outputs back—scenarios like Pascal’s Mugging are the equivalent of “suppose something really weird happens; wouldn’t that be weird?” to which the answer is “well, yes.”

The really nice thing about VNM is that it’s the

onlyrule that’s mathematically consistent with itself and a handful of nice axioms. You might give up one of those axioms, but for any of those axioms we can show an example where a rule that doesn’t follow that axiom will take sensible inputs and give silly outputs. So I don’t think there’s much to be gained by trying to replace a mean decision rule with a median decision rule or some other decision rule—but there is a lot to be gained by sharpening our probability distributions and more clearly figuring out our mapping from world-histories to utilities.To me, consequentialism is either something trivial or something I reject, but that said this is a fully general (and kind of weak) counter-argument. I can apply it to Newcomb from the CDT point of view. I can apply it to Smoking Lesion from the EDT point of view! I can apply it to astronomic data from the Ptolemy’s theory of celestial motion point of view! We have to deal with things!

I hesitate to call consequentialism trivial, because I wouldn’t use it to describe a broad class of ‘intelligent’ agents, but I also wouldn’t reject it, because it does describe the

designof those agents.I don’t see it as a counter-argument. In general, I think that a method is appropriate if hard things are hard and easy things are easy—and, similarly, normal things are normal and weird things are weird. If the output is weird in the same way that the input is weird, the system is behaving appropriately; if it adds or subtracts weirdness, then we’re in trouble!

For example, suppose you supplied a problem with a relevant logical contradiction to your decision algorithm, and it spat out a single numerical answer. Is that a sign of robustness, or lack of robustness?

I just meant I accept the consequentialist idea in decision theory that we should maximize, e.g. pick the best out of alternatives. But said in this way, it’s a trivial point. I reject more general varieties of consequentialism (for reasons that are not important right now, but basically I think a lot of weird conclusions of consequentialism are due to modeling problems, e.g. the set up that makes consequentialism work doesn’t apply well).

I don’t know what you are saying here. Can you taboo “weird?” Newcomb is weird for CDT because it explicitly violates an assumption CDT is using. The answer here is to go meta and think about a family of decision theories of which CDT is one, indexed by their assumption sets.

I understood and agree with that statement of consequentialism in decision theory—what I disagree with is that it’s trivial that maximization is the right approach to take! For many situations, a reflexive agent that does not actively simulate the future or consider alternatives performs better than a contemplative agent that does simulate the future and considerate alternatives, because the best alternative is “obvious” and the acts of simulation and consideration consume time and resources that do not pay for themselves.

That’s obviously what’s going on with thermostats, but I would argue is what goes on all the way up to the consequentialism-deontology divide in ethics.

I would probably replace it with Pearl’s phrase here, of “surprising or unbelievable.”

To use the specific example of Newcomb’s problem, if people find a perfect predictor “surprising or unbelievable,” then they probably

alsothink that the right thing to do around a perfect predictor is “surprising or unbelievable,” because using logic on an unbelievable premise can lead to an unbelievable conclusion! Consider a Mundane Newcomb’s problem which is missing perfect prediction but has the same evidential and counterfactual features: that is, Omega offers you the choice of one or two boxes, you choose which boxes to take, and then it puts a million dollars in the red box and a thousand dollars in the blue box if you choose only the red box and it puts a thousand dollars in the blue box if you choose the blue box or no boxes. Anyone that understands the scenario and prefers more money to less money will choose just the red box, and there’s nothing surprising or unbelievable about it.What is surprising is the claim that there’s an entity who can replicate the counterfactual structure of the Mundane Newcomb scenario

withoutalso replicating the temporal structure of that scenario. But that’s a claim about physics, not decision theory!Absolutely. This is the “bounded rationality” setting lots of people think about. For instance, Big Data is fashionable these days, and lots of people think about how we may do usual statistics business under severe computational constraints due to huge dataset sizes, e.g. stuff like this:

http://www.cs.berkeley.edu/~jordan/papers/blb_icml2012.pdf

But in bounded rationality settings we still want to pick the best out of our alternatives, we just have a constraint that we can’t take more than a certain amount of resources to return an answer. The (trivial) idea of doing your best is still there. That is the part I accept. But that part is boring, thinking of the right thing to maximize is what is very subtle (and may involve non-consequentialist ideas, for example a decision theory that handles blackmail may involve virtue ethical ideas because the returned answer depends on “the sort of agent” someone is).

I don’t agree. Utility is a separate concept from expected value maximization. Utility is a way of ordering and comparing different outcomes based on how desirable they are. You can say that one outcome is more desirable than another, or even quantify how many times more desirable it is. This is a useful and general concept.

Expected utility does have some nice properties being completely consistent. However I argued above that this isn’t a necessary property. It adds complexity, sure, but if you self modify your decision making algorithm or predetermine your actions, you can force your future self to be consistent with your present self’s desires.

Expected utility is perfectly rational as the number of “bets” you take goes to infinity. Rewards will cancel out the losses in the limit, and so any agent would choose to follow EU regardless of their decision making algorithm. But as the number of bets becomes finite, it’s less obvious that this is the most desirable strategy.

Pascal’s Mugging isn’t “weird”, it’s perfectly typical. There are probably an infinite number of pascal’s mugging type situations. Hypotheses with exceedingly low probability but high utility.

If we built an AI today, based on pure expected utility, it would most likely fail spectacularly. These low probability hypotheses would come to totally dominate it’s decisions. Perhaps it would start to worship various gods and practice rituals and obeying superstitions. Or something far more absurd we haven’t even thought of.

And if you really believe in EU, you can’t say that this behavior is wrong or undesirable. This is what you should be doing, if you could, and you are losing a huge amount of EU by not doing it. You should want more than anything in existence, the ability to exactly calculate these hypotheses so you can collect that EU.

I don’t want that though. I want a decision rule such that I am very likely to end up in a good outcome. Not one where I will mostly likely end up in a very suboptimal outcome, with an infinitesimal probability of winning the infinite utility lottery.

Expected utility is convenient and makes for a nice mathematical theory.

It also makes a lot of assumptions. One assumes that the expectation does, in fact, exist. It need not. For example, in a game where two players toss a fair coin, we expect that in the long run the number of heads should equal the number of tails at some point. It turns out that the expected waiting time is

infinite. Then there’s the classic St. Petersburg paradox.There are examples of “fair” bets (i.e. expected gain is 0) that are nevertheless unfavorable (in the sense that you’re almost certain to sustain a net loss over time).

Expected utility is a

modelof reality that does a good job in many circumstances but has some key drawbacks where naive application will lead to unrealistic decisions. The map is not the territory, after all.To Bentham, sure; today, we call something that generic “ranking” or something similar, because VNM-utility is the only game in town when it comes to assigning real-valued desirabilities to consequences.

Disagreed. The proof of the VNM axioms goes through for a single bet; I recommend you look that up, and then try to create a counterexample.

Note that it’s easy to come up with a

wrong utility mapping. One could, say, map dollars linearly to utility and then say “but I don’t prefer a half chance of $100 and half chance of nothing to a certain $50!” , but that’s solved by changing the utility mapping from linear from sublinear (say, log or sqrt or so on). In order to exhibit a counterexample it has to look like the Allais paradox, where someone confirms two preferences and then does not agree with the consequence of those preferences considered together.It probably isn’t the case that there are an infinite number of situations where the utility times the probability is higher than the cost, and if there are, that’s probably a faulty utility function or faulty probability estimator rather than a faulty EU calculation. Consider this bit from You and Your Research by Hamming:

An AI might correctly calculate that time travel is the most positive technology it could possibly develop—but also quickly calculate that it has no idea where to even start, and so the probability of success from thinking about it more is low enough that it should go for a more credible option. That’s what human thinkers do and it doesn’t seem like a mistake in the way that the Allais paradox seems like a mistake.

Pascal’s wager

isthe counterexample, and it’s older than VNM. EY’s Pascal’s mugging was just an attempt to formalize it a bit more and prevent silly excuses like “well what if we don’t allow infinites or assume the probabilities exactly cancel out.”Counterexample in that it violates what humans want, not that it produces inconsistent behavior or anything. It’s perfectly valid for an agent to follow EU, as it is for it to follow my method. What we are arguing about is entirely subjective.

If you really believe in EU a priori, then no argument should be able to convince you it is wrong. You would find nothing wrong with Pascal situations, and totally agree with the result of EU. You wouldn’t have to make clever arguments about the utility function or probability estimates to get out of it.

This is pretty thoroughly argued in the original Pascal’s Mugging post. Hypotheses of vast utility can grow much faster than their improbability. The hypothesis “you will be rewarded/tortured 3^^^3 units” is infinitesimally smaller in an EU calculation to the hypothesis “you will be rewarded/tortured 3^^^^^^^3 units”, and only takes a few more bits to express, and it can grow even further.

Counterexample in what sense? If you do in fact receive infinite utility from going to heaven, and being Christian raises the chance of you going to heaven by any positive amount over your baseline chance, then it is the right move to be Christian instead of baseline.

The reason people reject Pascal’s Wager or Mugging is, as I understand it, they don’t see the statement “you receive infinite utility from X” or “you receive a huge amount of disutility from Y” as actual evidence about their future utility.

In general, I think that any problem which includes the word “infinite” is guilty until proven innocent, and it is much better to express it as a limit. (This clears up a huge amount of confusion.) And the general principle- that as the prize for winning a lottery gets better, the probability of winning the lottery necessary to justify buying a fixed-price ticket goes down, seems like a reasonable principle to me.

I think money pumps argue against subjectivity. Basically, if you use an inconsistent decision theory, someone else can make money off your inconsistency or you don’t actually use that inconsistent decision theory.

I will say right now: I believe that if you have a complete set of outcomes with known utilities and the probabilities of achieving those outcomes conditioned on taking actions from a set of possible actions, the best action in that set is the one with the highest probability-weighted utility sum. That is, EU maximization works if you feed it the right inputs.

Do I think it’s trivial to get the right inputs for EU maximization? No! I’m not even sure it’s

possibleexcept in approximation. Any problem that starts with utilities in the problem description has hidden the hard work under the rug, and perhaps that means they’ve hidden a ridiculous premise.Assuming a particular method of assigning prior probabilities to statements, yes. But is that the right method of assigning prior probabilities to statements?

(That is, yes, I’ve read Eliezer’s post, and he’s asking how to generate probabilities of consequences given actions. That’s a

physicsquestion, not a decision theory question.)Where “right” is defined as “maximizing expected utility”, then yes. It’s just a tautology, “maximizing expected utility maximizes expected utility”.

My point is if you actually asked the average person, even if you explained all this to them, they would still not agree that it was the right decision.

There is no law written into the universe that says you

haveto maximize expected utility. I don’t think that’ what humans really want. If we choose to follow it, in many situation it will lead to undesirable outcomes. And it’s quite possible that those situations are actually common.It may mean life becomes more complicated than making simple EU calculations, but you can still be perfectly consistent (see further down.)

You could express it as a limit trivially (e.g. a hypothesis that in heaven you will collect 3^^^3 utilons per second for an unending amount of time.)

Sounds reasonable, but it breaks down in extreme cases, where you end up spending

almost all of your probability massin exchange for a single good future with arbitrarily low probability.Here’s a thought experiment. Omega offers you tickets for 2 extra lifetimes of life, in exchange for a 1% chance of dying when you buy the ticket. You are forced to just keep buying tickets until you finally die.

Maybe you object that you discount extra years of life by some function, so just modify the thought experiments so the reward increase factorially per ticket bought, or something like that.

Fortunately we don’t have to deal with these situations much, because we happen to live in a universe where there aren’t powerful agents offering us very high utility lotteries. But these situations occur all the time if you deal with hypotheses instead of lotteries. The only reason we don’t notice it is because we ignore or refuse to assign probability estimates to very unlikely hypotheses. An AI might not, and so it’s very important to consider this issue.

My method isn’t vulnerable to money pumps, as is an infinite number of arbitrary algorithms of the same class. See my comment here for details.

You don’t even need the stuff I wrote about predetermining actions, that just minimizes regret. Even a naive implementation of expected median utility should not be money pumpable.

The method by which you assign probabilities should be unrelated to the method you assign utilities to outcomes. That is, you can’t just say you don’t like the outcome EU gives you and so assign it a lower probability, that’s a horrible violation of Bayesian principles.

I don’t know what the correct method of assigning probabilities, but even if you discount complex hypotheses factorially or something, you still get the same problem.

I certainly think these scenarios have reasonable prior probability. God could exist, we could be in the matrix, etc. I give them so low probability I don’t typically think about them, but for this issue that is irrelevant.

Yes. That’s the thing that sounds silly but is actually deep.

That is the objection, but I think I should explain it in a more fundamental way.

What is the utility of a consequence? For simplicity, we often express it as a real number, with the caveat that all utilities involved in a problem have their relationships preserved by an affine transformation. But that number is

groundedby a gamble. Specifically, consider three consequences, A, B, and C, with u(A)<u(B)<u(C). If I amindifferentbetween B for certain and A with probabilitypand C otherwise, I encode that with the mathematical relationship:As I express more and more preferences, each number is grounded by more and more constraints.

The place where counterexamples to EU calculations go off the rails is when people intervene at the intermediate step. Suppose

pis 50%, and I’ve assigned 0 to A, 1 to B, and 2 to C. If a new consequence, D, is introduced with a utility of 4, that immediately implies:I am indifferent between (50% A, 50% D) and (100% C).

I am indifferent between (75% A, 25% D) and (100% B).

I am indifferent between (67% B, 33% D) and (100% C).

If one of those three statements is not true, I can use that and D having a utility of 4 to prove a contradiction. But the while the existence of D and my willingness to accept those specific gambles implies that D’s utility is 4, the existence of the number 4

does notimply that there exists a consequence where I’m indifferent to those gambles!And so very quickly Omega might have to offer me a lifetime longer than the lifetime of the universe, and because I don’t believe that’s possible I say “no thanks, I don’t think you can deliver, and in the odd case where you

candeliver, I’m not sure that I want what you can deliver.” (This is the resolution of the St. Petersburg Paradox where you enforce that the house cannot pay you more than the total wealth of the Earth, in which case the expected value of the bet comes out to a reasonable, low number of dollars, roughly where people estimate the value of the bet.)To me, this maps on to basic research. There’s some low probability that a particular molecule cures a particular variety of cancer, but it would be great if it did—so let’s check. The important conceptual addition from this analogy is that both

hypotheses are entangled(this molecule curing cancer implies things about other molecules) andvalues are entangled(the second drug that can cure a particular variety of cancer is less valuable than the first drug that can).And so an AI might have somewhat different basic research priorities than we do—indeed, humans vary widely on their preferences for following various uncertain paths—but it seems likely to me that it could behave reasonably when coming up with a portfolio of actions to take, even if it looks like it might behave oddly with only one option.

Playing against nature, sure—but playing against an intelligent agent? Running a minimax calculation to figure out that my opponent is not likely to let me checkmate him in one move is hardly a horrible violation of Bayesian principles!

This is a cop out. Obviously that

specificsituation can’t occur in reality, that’s not the point. If your decision algorithm fails in some extreme cases, at least confess that it’s not universal.Same thing. Omega’s ability and honesty are premises.

The point of the thought experiment is just to show that EU is required to trade away huge amounts of outcome-space for really good but improbable outcomes. This is a good strategy if you plan on making an infinite number of bets, but horrible if you don’t expect to live forever.

I don’t get your drug research analogy. There is no pascal’s equivalent situation in drug research. At best you find a molecule that cures

alldiseases, but that’s hardly infinite utility.Instead it would be more like, “there is a tiny, tiny probability that a virus could emerge which causes humans not to die, but to suffer for eternity in the worst pain possible. Therefore, by EU calculation, I should spend all of my resources searching for a possible vaccine for this specific disease, and nothing else.”

What does it mean for a decision algorithm to fail? I’ll give an answer later, but here I’ll point out that I

doendorse that multiplication of reals is universal—that is, I don’t think multiplication breaks down when the numbers get extreme enough.And an unbelievable premise leads to an unbelievable conclusion. Don’t say that

logichas broken down because someone gave the syllogism “All men are immortal, Socrates is a man, therefore Socrates is still alive.”How does [logic work]? Eliezer puts it better than I can:

EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to

producepreferences but topreservethem. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.And if it’s

nottrue that you would trade away huge amounts of outcome-space for really good but improbable outcomes, this is a fact about your preference model that EU preserves! Remember, EU preference models map lists of outcomes to classes of lists of real numbers, but the inverse mapping is not guaranteed to have support over the entire reals.I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm, and the chief ways to do so are 1) to do the math wrong and be inconsistent and 2) to make it more costly to express your preferences or world-model.

We have lots of hypotheses about low-probability, high-payout options, and if humans make mistakes, it is probably by

overestimatingthe probability of low-probability events andoverestimatinghow much we’ll enjoy the high payouts, both of which make us more likely to pursue those paths than a rational version of ourselves.So it seems to me that if we have an algorithm that can correctly manage the budget of a pharmaceutical corporation, balancing R&D and marketing and production and so on, that requires solving this philosophical problem. But we

havethe mathematical tools to correctly manage the budget of a pharmaceutical corporation given a world-model, which says to me we should turn our attention to getting more precise and powerful world-models.When it makes decisions that are undesirable. There is no point deciding to run a decision algorithm which is perfectly consistent but results in outcomes you don’t want.

In the case of the Omega’s-life-tickets scenario, one could argue it fails in an objective sense since it will never stop buying tickets until it dies. But that wasn’t even the point I was trying to make.

I don’t know if there is a name for this fallacy but there should be. Where someone objects to the premises of a hypothetical situation intended just to demonstrate a point. E.g. people who refuse to answer the trolley dilemma and instead say “but that will probably never happen!” It’s very frustrating.

This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.

What I’ve been saying from the very beginning is that there isn’t any reason to believe there is

anyutility function that will produce desirable outcomes if fed to an expected utility maximizer.Even if you are an EU maximizer, EU will make you “predictably” worse off, as in the majority of cases you will be worse off. A true EU maximizer doesn’t care so long as the utility of the very low probability outcomes is high enough.

One name is fighting the hypothetical, and it’s worth taking a look at the least convenient possible world and the true rejection as well.

There are good and bad reasons to fight the hypothetical. When it comes to these particular problems, though, the objections I’ve given

aremy true objections. The reason I’d only pay a tiny amount of money for the gamble in the St. Petersburg Paradoxisthat there is only so much financial value that the house can give up. One of the reasons I’m sure this is my true objection is because the richer the house, the more I would pay for such a gamble. (Because there are no infinitely rich houses, there is no one I would pay an infinite amount to for such a gamble.)I’m not sure why you think it’s

subtle—I started off this conversation with:But I don’t think it’s quite right to call it “circular,” for roughly the same reasons I don’t think it’s right to call

logic“circular.”To make sure we’re talking about the same thing, I think an expected utility maximizer (EUM) is something that takes both a function u(O) that maps outcomes to utilities, a function p(A->O) that maps actions to probabilities of outcomes, and a set of possible actions, and then finds the action out of all possible A that has the maximum weighted sum of u(O)p(A->O) over all possible O.

So far, you

have notbeen arguing thatevery possibleEUM leads to pathological outcomes; you have been exhibiting particular combinations of u(O) and p(A->O) that lead to pathological outcomes, and I have been responding with “have you tried not using those u(O)s and p(A->O)s?”.It doesn’t seem to me that this conversation is producing value for either of us, which suggests that we should either restart the conversation, take it to PMs, or drop it.

This suggests buying tickets takes finite time per ticket, and that the offer is perpetually open. It seems like you could get a solid win out of this by living your life, buying one ticket every time you start running out of life. You keep as much of your probability mass alive as possible for as long as possible, and your probability of being alive at any given time after the end of the first “lifetime” is greater than it would’ve been if you hadn’t bought tickets. Yeah, Omega has to follow you around while you go about your business, but that’s no more obnoxious than saying you have to stand next to Omega wasting decades on mashing the ticket-buying button.

Ok change it so the ticket booth closes if you leave.

That’s not the way in which maximizing expected utility is perfectly rational.

The way it’s perfectly rational is this. Suppose you have any decision making algorithm; if you like, it can have an internal variable called “utility” that lets it order and compare different outcomes based on how desirable they are. Then either:

the algorithm has some ugly behavior with respect to a

finitecollection of bets (for instance, there are three bets A, B, and C such that it prefers A to B, B to C, and C to A), orthe algorithm is equivalent to one which maximizes the expected value of

someutility function: maybe the one that your internal variable was measuring, maybe not.The first condition is not true, since it gives a consistent value to any probability distribution of utilities. The second condition is not true other since the median function is not merely a transform of the mean function.

I’m not sure what the “ugly” behavior you describe is, and I bet it rests on some assumption that’s too strong. I already mentioned how inconsistent behavior can be fixed by allowing it to predetermine it’s actions.

You can find the Von Neumann—Morgenstern axioms for yourself. It’s hard to say whether or not they’re too strong.

The problem with “allowing [the median algorithm] to predetermine its actions” is that in this case, I no longer know what the algorithm outputs in any given case. Maybe we can resolve this by considering a case when the median algorithm fails, and you can explain what your modification does to fix it. Here’s an example.

Suppose I roll a single die.

Bet A loses you $5 on a roll of 1 or 2, but wins you $1 on a roll of 3, 4, 5, or 6.

Bet B loses you $5 on a roll of 5 or 6, but wins you $1 on a roll of 1, 2, 3, or 4.

Bet A has median utility of U($1), as does bet B. However, combined they have a median utility of U(-$4).

So the straightforward median algorithm pays money to buy Bet A, pays money to buy Bet B, but will then pay money to be rid of their combination.

I think I’ve found the core of our disagreement. I want an algorithm that considers all possible paths through time. It decides on a set of actions, not just for the current time step, but for all possible future time steps. It chooses such that the final probability distribution of possible outcomes, at some point in the future, is optimal according to some metric. I originally thought of median, but it can work with any arbitrary metric.

This is a

generalization ofexpected utility. The VNM axioms require an algorithm to make decisions independently and Just In Time. Whereas this method lets it consider all possible outcomes. It may be less elegant than EU, but I think it’s closer to what humans actually want.Anyway your example is wrong, even without predetermined actions. The algorithm would buy bet A, but then not buy bet B. This is because it doesn’t consider bets in isolation like EU, but considers it’s entire probability distribution of possible outcomes. Buying bet B would decrease it’s expected median utility, so it wouldn’t take it.

No, they don’t.

Assuming the bet has a fixed utility, then EU gives it a fixed estimate right away. Whereas my method considers it along with all other bets that it’s made or expects to make, and it’s estimate can change over time. I should have said that it’s not independent or

fixed, but that is what I meant.In the VNM scheme where expected utility is derived at a consequence of the axioms, the way that a bet’s utility changes over time is that its utility is not fixed. Nothing at all stops you from changing the utility you attach to a 50:50 gamble of getting a kitten versus $5 if your utility for a kitten (or for $5) changes: for example, if you get another kitten or win the lottery.

Generalizing to allow the value of the bet to change when the value of the options did

notchange seems strange to me.I am lost, this is just EU in a longitudinal setting? You can average over lots of stuff. Maximizing EU is boring, it’s specifying the right distribution that’s tricky.

It’s not EU, since it can implement arbitrary algorithms to specify the desired probability distribution of outcomes. Averaging utility is only one possibility, another I mentioned was median utility.

So you would take the median utility of all the possible outcomes. And then select the action (or series of actions in this case) that leads to the highest median utility.

No method of specifying utilities would let EU do the same thing, but you can trivially implement EU in it, so it’s strictly more general than EU.

So, I think you might be interested in UDT. (I’m not sure what the current best reference for that is.) I think that this requires actual omniscience, and so is not a good place to look for decision algorithms.

(Though I should add that typically utilities are defined over world-histories, and so any decision algorithm typically identifies classes of ‘equivalent’ actions, i.e. acknowledges that this is a thing that needs to be accepted somehow.)

UDT is overkill. The idea that all future choices can be collapsed into a single choice appears in the work of von Neumann and Morgenstern, but is probably much older.

Oh, I see. I didn’t take that problem into account, because it doesn’t matter for expected utility, which is additive. But you’re right that considering the entire probability distribution is the right thing to do, and under than assumption we’re forced to be transitive.

The actual VNM axiom violated by median utility is independence: If you prefer X to Y, then a gamble of X vs Z is preferable to the equivalent gamble of Y vs Z. Consider the following two comparisons:

Taking bet A, as above, versus the status quo.

A

^{2}⁄_{3}chance of taking bet A and a^{1}⁄_{3}chance of losing $5, versus a^{2}⁄_{3}chance of the status quo and a^{1}⁄_{3}chance of losing $5.In the first case, bet A has median utility U($1) and the status quo has U($0), so you pick bet A. In the second case, a gamble with a possibility of bet A has median utility U(-$5) and a gamble with a possibility of the status quo still has U($0), so you pick the second gamble.

Of course, independence is probably the shakiest of the VNM axioms, and it wouldn’t surprise me if you’re unconvinced by it.