Against Discount Rates

Eliezer Yudkowsky21 Jan 2008 10:00 UTC

37 points

I’ve never been a fan of the notion that we should (normatively) have a discount rate in our pure preferences—as opposed to a pseudo-discount rate arising from monetary inflation, or from opportunity costs of other investments, or from various probabilistic catastrophes that destroy resources or consumers. The idea that it is literally, fundamentally 5% more important that a poverty-stricken family have clean water in 2008, than that a similar family have clean water in 2009, seems like pure discrimination to me—just as much as if you were to discriminate between blacks and whites.

And there’s worse: If your temporal discounting follows any curve other than the exponential, you’ll have time-inconsistent goals that force you to wage war against your future selves—preference reversals—cases where your self of 2008 will pay a dollar to ensure that your future self gets option A in 2011 rather than B in 2010; but then your future self in 2009 will pay another dollar to get B in 2010 rather than A in 2011.

But a 5%-per-year discount rate, compounded exponentially, implies that it is worth saving a single person from torture today, at the cost of 168 people being tortured a century later, or a googol persons being tortured 4,490 years later.

People who deal in global catastrophic risks sometimes have to wrestle with the discount rate assumed by standard economics. Is a human civilization spreading through the Milky Way, 100,000 years hence—the Milky Way being about 100K lightyears across—really to be valued at a discount of 10^-2,227 relative to our own little planet today?

And when it comes to artificial general intelligence… I encounter wannabe AGI-makers who say, “Well, I don’t know how to make my math work for an infinite time horizon, so… um… I’ve got it! I’ll build an AGI whose planning horizon cuts out in a thousand years.” Such a putative AGI would be quite happy to take an action that causes the galaxy to explode, so long as the explosion happens at least 1,001 years later. (In general, I’ve observed that most wannabe AGI researchers confronted with Singularity-level problems ponder for ten seconds and then propose the sort of clever programming trick one would use for data-mining the Netflix Prize, without asking if it makes deep sense for Earth-originating civilization over the next million years.)

The discount question is an old debate in economics, I know. I’m writing this blog post just now, because I recently had a conversation with Carl Shulman, who proposed an argument against temporal discounting that is, as far as I know, novel: namely that an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel—maybe hunting for wormholes with a terminus in the past.

Or to translate this back out of transhumanist discourse: If you wouldn’t burn alive 1,226,786,652 people today to save Giordano Bruno from the stake in 1600, then clearly, you do not have a 5%-per-year temporal discount rate in your pure preferences.

Maybe it’s easier to believe in a temporal discount rate when you—the you of today—are the king of the hill, part of the most valuable class of persons in the landscape of present and future. But you wouldn’t like it if there were other people around deemed more valuable than yourself, to be traded off against you. You wouldn’t like a temporal discount if the past was still around.

Discrimination always seems more justifiable, somehow, when you’re not the person who is discriminated against -

- but you will be.

(Just to make it clear, I’m not advocating against the idea that Treasury bonds can exist. But I am advocating against the idea that you should intrinsically care less about the future than the present; and I am advocating against the idea that you should compound a 5% discount rate a century out when you are valuing global catastrophic risk management.)

What links here?

Eliezer Yudkowsky21 Jan 2008 10:00 UTC

37 points

81 comments2 min readLW link Archive

Utility Functions

Paul_Crowley 21 Jan 2008 11:33 UTC
−1 points
Obviously there’s another sort of discounting that does make sense. If you offer me a choice of a dollar now or $1.10 in a year, I am almost certain you will make good on the dollar now if I accept it, whereas there are many reasons why you might fail to make good on the $1.10. This sort of discounting is rationally hyperbolic, and so doesn’t lead to the paradoxes of magnitude over time that you highlight here.
- gwern 21 Jan 2011 22:42 UTC
  10 points
  Parent
  Yes, that discounting makes sense, but it’s explicitly not what Eliezer is talking about. His very first sentence:
  
  “I’ve never been a fan of the notion that we should (normatively) have a discount rate in our pure preferences—as opposed to a pseudo-discount rate arising from monetary inflation, or from opportunity costs of other investments, or from various probabilistic catastrophes that destroy resources or consumers.”
  
  (Also, I don’t see how that example is ‘hyperbolic’.)
  - Perplexed 21 Jan 2011 23:11 UTC
    0 points
    Parent
    
    Also, I don’t see how that example is ‘hyperbolic’.
    
    Agree. Not hyperbolic.
    
    Assuming, in Paul Crowley’s example, that there is a constant rate of failure (conditional on not having already failed), this yields well-behaved exponential discounting, which is relatively paradox-free.
- waveman 19 Mar 2014 4:49 UTC
  0 points
  Parent
  Good point.
  
  More generally as per the wikipedia article http://en.wikipedia.org/wiki/Hyperbolic_discounting#Criticism exponential discounting is only correct if you are equally certain of the payoffs at all the different times.
  
  More broadly it assumes no model error. Whatever decision model you are using you need to be 100% certain of it to justify exponential discounting.
  
  Nassim Taleb points out that quite a few alleged biases are actually quite rational when taking into account model error and he includes a derivation of why the hyperbolic discounting formula is actually valid in many situations.
  
  Silent Risk Section 4.6 Psychological pseudo-biases under second layer of uncertainty. Draft at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2392310
tcpkac 21 Jan 2008 12:03 UTC
0 points
Put baldly, the main underlying question is : how do you compare the value of (a) a unit of work expended now, today, on the well-being of a person alive, now, today, with the value of (b) the same unit of work expended now, today, for the well-being of 500 potential people who might be alive in 500 years’ time, given that units of work are in limited supply. I suspect any attempt at a mathematical answer to that would only be an expression of a subjective emotional preference. What is more, the mathematical answer wouldn’t be a discount function, it would be a compounding function, as it would be the result of comparing all the AI units of work available between now and time t in the future, with the units of work required between now and time t to address all the potential needs of humanity and trans-humanity between now and the end of time, which looks seriously like infinity.
RobinHanson 21 Jan 2008 13:51 UTC
12 points
0
You might need a time machine to give a better experience to someone long dead, but not to give them more of what they wanted. For example, if they wanted to be remembered and revered, we can do that for them today. But we don’t do much of that, for them. So we don’t need time machines to see we don’t care that much about our ancestors. We do in fact have conflicts across time, where we each time would prefer allocate resources differently. That is why we should try to arrange deals across time, where for example, we agree to invest for the future, and they agree to remember and revere us.
Vladimir_Nesov2 21 Jan 2008 14:17 UTC
0 points
Discount rate takes care of effect your effort can have on the future, relative to effect it will have on present, it has nothing to do with ‘intrinsic utility’ of things in the future. Future doesn’t exist in the present, you only have a model of the future when you make decisions in the present. Your current decisions are only as good as you can anticipate their effect in the future, and process Robin described in his blog post replay is how it can proceed, it assumes that you know very little and will be better off with just passing resources to future folk to take care of whatever they need themselves.
Silas 21 Jan 2008 14:18 UTC
2 points
My first reaction is to guess that people now are “worth” more than people in 1600 because they have access to more productivity-enhancing equipment, including life-extending equipment. So a proper accounting would make it more like 6000 people. Furthermore, more productivity from someone in the year 1600 would facilitate exponentially more resources (including life-saving resource) over the time since, saving more than 6000 people. After all, that’s why interest exists—because the forgone opportunity grows exponentially! So, even valuing the people equally may justify sacrificing the 12 million for the one.

But I freely admit I may need to rethink that.

Also,

an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel—maybe hunting for wormholes with a terminus in the past.

There is a financial argument against the possibility time travel that was published in a journal (don’t have the citation offhand): if it were possible to time-travel, interest rates would be arbitraged to zero. Realizing this, wouldn’t the AI give up on that goal? [/possibly naive]
- SilasBarta 23 Sep 2009 17:28 UTC
  −8 points
  Parent
  w00t! Got a pre-LW comment of mine modded up!
CarlShulman 21 Jan 2008 14:46 UTC
1 point
Robin,

“That is why we should try to arrange deals across time, where for example, we agree to invest for the future, and they agree to remember and revere us.” Consider an agent that at any time t does not discount benefits received between t and t+1 year, discounts benefits between t+1 years and t+100 years by half, and does not value benefits realized after t+100 years. If the agent is capable of self-modification, then at any particular time it will want to self-modify to replace the variable ‘t’ with a constant, the time of self-modification, locking in its preferences over world histories for its future selves. The future selves will then expend all available resources in rushed consumption over the next 100 years. So I would expect the bargaining position of the future to get progressively worse with advancing technology.
Peter_de_Blanc 21 Jan 2008 16:27 UTC
1 point
Eli said: I encounter wannabe AGI-makers who say, “Well, I don’t know how to make my math work for an infinite time horizon, so… um… I’ve got it! I’ll build an AGI whose planning horizon cuts out in a thousand years.”

I’m not sure if you’re talking about me. I have said that I think we need some sort of bounded utility function, but that doesn’t mean it has to be an integral of discounted time-slice values.
Eliezer Yudkowsky 21 Jan 2008 18:15 UTC
11 points
Peter, it wasn’t just you, it was Marcus Hutter’s AIXI formalism, and I think at least one or two other people.

Nonetheless, what you proposed was indeed a grave sin. If your own utility function is not bounded, then don’t build an AI with a bounded utility function, full stop. This potentially causes infinite damage. Just figure out how to deal with unbounded utility functions. Just deal, damn it.

Of all the forms of human possibility that you could destroy in search of cheap math, going from infinite potential to finite potential has to be one of the worst.
- timtyler 16 Mar 2011 20:11 UTC
  2 points
  Parent
  I don’t think the human brain’s equivalent to a utility function is unbounded. Dopamine levels and endorphin levels are limited—and it seems tremendously unlikely that the brain deals with infinities in its usual mode of operation. So, this is all very hypothetical.
  - JoshuaZ 16 Mar 2011 21:15 UTC
    9 points
    Parent
    
    I don’t think the human brain’s equivalent to a utility function is unbounded. Dopamine levels and endorphin levels are limited—and it seems tremendously unlikely that the brain deals with infinities in its usual mode of operation. So, this is all very hypothetical.
    
    This doesn’t have much to do with my preferences. I might experience the same level of negative emotion when thinking about Busy Beaver (10) people being tortured as opposed to Busy Beaver(1000) people being tortured, but I still have a preference for which one I’d like if I have to choose between the two.
    - timtyler 16 Mar 2011 22:12 UTC
      −3 points
      Parent
      Some mechanism in your (finite) brain is still making that decision.
      - JoshuaZ 16 Mar 2011 22:23 UTC
        4 points
        Parent
        
        Some mechanism in your (finite) brain is still making that decision.
        
        Sure. But I can express a preference about infinitely many cases in a finite statement. In particular, my preferences includes something like the following: given the existence of k sentient, sapient entities, and given i < j ⇐ k, I prefer i entities getting tortured to j entities getting tortured assuming everything else is otherwise identical.
        timtyler 16 Mar 2011 22:42 UTC
        −8 points
        Parent
        Alas, your brain can’t handle those numbers—beyond a certain point. They can’t even be input into your brain in your lifetime.
        
        If we are talking about augmenting your brain with a machine, so it is able to deal with these huge numbers, those aren’t really the preferences of a human being any more—and you still don’t get to “unbounded” in a finite time—due to the finite visible universe.
        
        I’m not sure how utility (and expected utility) are physically represented in the human brain. Dopamine levels and endorphin levels are the most obvious candidates, but there are probably also various proxies. However, I figure a 16-bit number would probably cover it pretty well. It may seem counter-intuitive—but you don’t really need more than that to make decisions of the type you describe—even for numbers of people with (say) 256-bit representations.
        benelliott 16 Mar 2011 23:16 UTC
        9 points
        Parent
        Think about it this way:
        
        Omega comes up to you and offers you a choice, it will kill either n or 2n people depending on what you ask. When you ask what n is Omega explains that it is an integer, but is unfortunately far too large to define it within your lifetime. Would you not still pick n in this dilemma? I know I would.
        
        This isn’t quite enough to prove an unbounded utility function, but if we slightly modify the choice so it is n people die with certainty versus 2n people die with 99.999% probability and nobody dies with 0.001% probability then it is enough.
        timtyler 16 Mar 2011 23:25 UTC
        −4 points
        Parent
        Your brain could probably make that kind of decision with only a few bits of utility. The function would go: lots-of-death bad, not-so-much-death not so bad. IMO, in no way is that evidence that the brain represents unbounded utilities.
        JGWeissman 16 Mar 2011 23:42 UTC
        4 points
        Parent
        
        The function would go: lots-of-death bad, not-so-much-death not so bad.
        
        Try using numbers. If you try to bound the function, there will be a sufficiently large n where you will prefer the 99.999% probability of 2n people dying to 100% of n people dying.
        timtyler 16 Mar 2011 23:47 UTC
        −3 points
        Parent
        To recap, I objected: “your brain can’t handle those numbers”. To avoid the huge numbers, they were replaced with “n”—and a bizarre story about an all-knowing being. If you go back to the numbers, we are back to the first objection again—there are some numbers that are too big for unmodified humans to handle. No, I can’t tell you which numbers—but they are out there.
        CuSithBell 17 Mar 2011 0:45 UTC
        6 points
        Parent
        The grandparent is a reductio of your assertion (and thus, if you agree that “not-so-much-death is not so bad”, a disproof). You seem to be questioning the validity of algebra rather than retracting the claim. Do you have a counterargument?
        
        I’d suggest that you may be able to argue that the brain does not explicitly implement a utility function as such, which makes sense because utility functions are monstrously complex. Instead, the brain likely implements a bunch of heuristics and other methods of approximating / instantiating a set of desires that could hypothetically be modeled by a utility function (that is unbounded).
        Expand this thread
        timtyler 17 Mar 2011 1:08 UTC
        0 points
        Parent
        
        The grandparent is a reductio of your assertion (and thus, if you agree that “not-so-much-death is not so bad”, a disproof). You seem to be questioning the validity of algebra rather than retracting the claim. Do you have a counterargument?
        
        “your brain can’t handle those numbers” wasn’t “questioning the validity of algebra”. It was questioning whether the human brain can represent—or even receive—the large numbers in question.
        CuSithBell 17 Mar 2011 1:20 UTC
        0 points
        Parent
        What you said was:
        
        To avoid the huge numbers, they were replaced with “n”
        
        as though this were somehow an indictment of the argument.
        
        Anyway, the important thing is: several people have already explained how a finite system can express an unbounded utility function without having to explicitly express numbers of unbounded size.
        timtyler 17 Mar 2011 1:46 UTC
        −5 points
        Parent
        
        What you said was:
        
        To avoid the huge numbers, they were replaced with “n”
        
        as though this were somehow an indictment of the argument.
        
        Dragging in Omega to represent the huge quantities for the human seems to have been a desperate move.
        
        Anyway, the important thing is: several people have already explained how a finite system can express an unbounded utility function without having to explicitly express numbers of unbounded size.
        
        Well, that’s OK—but the issue is what shape the human utility function is. You can’t just extrapolate out to infinity from a small number of samples near to the origin!
        
        I think there are limits to human happiness and pain—and whatever else you care to invoke as part of the human utility function—so there’s actually a finite representation with bounded utility—and I think that it is the best approximation to what the brain is actually doing.
        Strange7 17 Mar 2011 3:17 UTC
        −5 points
        Parent
        
        You can’t just extrapolate out to infinity from a small number of samples near to the origin!
        
        Some people can. It’s called proof by induction.
        JoshuaZ 17 Mar 2011 3:43 UTC
        6 points
        Parent
        
        You can’t just extrapolate out to infinity from a small number of samples near to the origin!
        
        Some people can. It’s called proof by induction.
        
        This is not how proof by induction works.
        Strange7 17 Mar 2011 7:23 UTC
        −7 points
        Parent
        If you think the proof is flawed, find a counterexample.
        
        A real, independently-verifiable counterexample, not just a nebulous spot on the number line where a counterexample might exist.
        CuSithBell 17 Mar 2011 13:16 UTC
        8 points
        Parent
        The proof by induction is correct. “Extrapolating from a small number of samples”, however, is not proof by induction.
        timtyler 17 Mar 2011 10:41 UTC
        −2 points
        Parent
        A fallible process. Pain might seem proportional to number of lashes at first—but keep going for a while, and you will see that they have a non-linear relationship.
  - wnoise 16 Mar 2011 23:07 UTC
    0 points
    Parent
    Dopamine levels and endorphin levels are not utility functions. At best they are “hedons”, and even that’s not indisputably clear—there’s more to happiness than that.
    
    A utility function is itself not something physical. It is one (often mathematically convenient) way of summarizing an agent’s preferences in making decisions. These preferences are of course physical. Note, for instance, that everything observable is completely invariant under arbitrary positive affine transformations. Even assuming our preferences can be described by a utility function (i.e. they are consistent—but we know they’re not), it’s clear that putting an upper bound on it would no longer agree with the decisions made by a utility function without such a bound.
    - timtyler 16 Mar 2011 23:31 UTC
      0 points
      Parent
      
      Dopamine levels and endorphin levels are not utility functions. At best they are “hedons”, and even that’s not indisputably clear—there’s more to happiness than that.
      
      Well, the brain represents utility somehow, as part of its operation. It rather obviously compares expected utilities of future states.
      
      I didn’t say dopamine levels and endorphin levels were utility functions. The idea is that they are part of the brain’s representation of expected utility—and utility.
      - wnoise 17 Mar 2011 8:15 UTC
        3 points
        Parent
        
        Well, the brain represents utility somehow, as part of its operation. It rather obviously compares expected utilities of future states.
        
        No. You’ve entirely missed my point. The brain makes decisions. Saying it does so via representing things as utilities is a radical and unsupported assumption. It can be useful to model people as making decisions according to a utility function, as this can compress our description of it, often with only small distortions. But it’s still just a model. Unboundedness in our model of a decision maker has nothing to do with unboundedness in the decision maker we are modeling. This is a basic map/territory confusion (or perhaps advanced: our map of their map of the territory is not the same as their map of the territory).
        timtyler 17 Mar 2011 10:33 UTC
        −1 points
        Parent
        Not exactly an assumption. We can see—more-or-less—how the fundamental reward systems in the brain work. They use neurotransmitter concentrations and firing frequencies to represent desire and and aversion—and pleasure and pain. These are the physical representation of utility, the brain’s equivalent of money. Neurotransmitter concentrations and neuron firing frequencies don’t shoot off to infinity. They saturate—resulting in pleasure and pain saturation points.
        FAWS 17 Mar 2011 10:45 UTC
        4 points
        Parent
        I see little indication that the brain is in the assigning absolute utilities business at all. Things like scope insensitivity seem to suggest that it only assigns relative utilities, comparing to a context-dependent default.
        rwallace 17 Mar 2011 13:24 UTC
        0 points
        Parent
        They are feedback signals, certainly. Every system with any degree of intelligence must have those. But feedback signals, utility and equivalent of money are not synonyms. To say a system’s feedback signals are equivalent to money is to make certain substantive claims about its design. (e.g. some but not most AI programs have been designed with those properties.) To say they are utility measurements is to make certain other substantive claims about its design. Neither of those claims is true about the human brain in general.
      - [deleted] 16 Mar 2011 23:58 UTC
        2 points
        Parent
        You argued that human utility is bounded because dopamine is bounded, and dopamine is part of how utility is represented. Yes? The obvious objection to your argument is that the representation could in principle take one of many different forms, some of which allow us to represent something unbounded by means of something bounded. If that were the case, then the boundedness of dopamine would not imply the boundedness of utility.
        
        If you want an example of how this representation might be done, here’s one: if you prefer state A to state B, this is (hypothetically) represented by the fact that if you move from state B to state A your dopamine level is raised temporarily—and after some interval, it drops again to a default level. So, every time you move from a less preferred state to a more preferred state, i.e. from lower utility to higher utility, your dopamine level is raised temporarily and then drops back. The opposite happens if you move from higher utility to lower utility.
        
        Though I have offered this as a hypothetical, from the little bit that I’ve read in the so-called “happiness” literature, something like this seems to be what actualyl goes on. If you receive good fortune, you get especially happy for a bit, and then you go back to a default level of happiness. And conversely, if you suffer some misfortune, you become unhappy for a bit, and then you go back to a default level of happiness.
        
        Unfortunately, a lot of people seem to draw what I think is a perverse lesson from this phenomenon, which is that good and bad fortune does not really matter, because no matter what happens to us, in the end we find ourselves at the default level of happiness. In my view, utility should not be confused with happiness. If a man becomes rich and, in the end, finds himself no happier than before, I don’t think that that is a valid argument against getting rich. Rather, temporary increases and decreases in happiness is how our brains mark permanent increases and decreases in utility. That the happiness returns to default does not mean that utility returns to default.
        timtyler 17 Mar 2011 0:03 UTC
        −3 points
        Parent
        
        You argued that human utility is bounded because dopamine is bounded, and dopamine is part of how utility is represented. Yes?
        
        No. What I actually said was:
        
        The idea is that they [Dopamine levels and endorphin levels] are part of the brain’s representation of expected utility—and utility.
        
        I do think an unbounded human-equivalent utility function is not supported by any evidence. I reckon Hutter’s [0,1] utility would be able to simulate humans just fine on digital hardware.
        [deleted] 17 Mar 2011 0:16 UTC
        3 points
        Parent
        I didn’t say that you equated utility with dopamine. [edit: I was replying to an earlier draft of your comment. As of now you’ve changed the comment to delete the claim that I had said that you equated utility with dopamine, though you retained an unexplained “no”.] I said that you said that dopamine is part of how utility is represented. Your quote appears to confirm my statement. You quote yourself saying “[Dopamine levels and endorphin levels] are part of the brain’s representation of expected utility—and utility.” Among other things, this says that dopamine is part of the brain’s representation of utility. Which is virtually word for word what I said you said, the main difference being that instead of saying “the brain’s representation of utility”, I said, “how utility is represented”. I don’t see any real difference here—just slightly different wording.
        
        Moreover, the key statement that I am basing my interpretation on is not that, but this:
        
        I don’t think the human brain’s equivalent to a utility function is unbounded. Dopamine levels and endorphin levels are limited—and it seems tremendously unlikely that the brain deals with infinities in its usual mode of operation. So, this is all very hypothetical.
        
        Here you are arguing that the human brain’s equivalent to a utility function is bounded, and your apparent argument for this is that dopamine and endorphin levels are limited.
        
        I argued that the limitation of dopamine and endorphin levels does not imply that the human brain’s equivalent to a utility function is bounded. You have not addressed my argument, only claimed—incorrectly, it would appear—that I had misstated your argument.
        timtyler 17 Mar 2011 0:23 UTC
        −1 points
        Parent
        I note that your characterisation of my argument models very, very poorly all the times I talked about the finite nature of the human brain on this thread.
        [deleted] 17 Mar 2011 0:32 UTC
        1 point
        Parent
        You are seriously referring me to your entire oeuvre as a supposed explanation of what you meant in the specific comment that I was replying to?
        timtyler 17 Mar 2011 0:58 UTC
        0 points
        Parent
        I was pointing out that there was more to the arguments I have given than what you said. The statement you used to characterise my position was a false syllogism—but it doesn’t represent my thinking on the topic very well.
  - benelliott 16 Mar 2011 21:59 UTC
    0 points
    Parent
    
    it seems tremendously unlikely that the brain deals with infinities in its usual mode of operation.
    
    Unbounded is not the same as infinite. The integers are unbounded but no integer is infinite. In the same way I can have a utility function with no upper bound on the values it outputs without it ever having to output infinity.
    - timtyler 16 Mar 2011 22:09 UTC
      −7 points
      Parent
      The human brain is limited to around 1,300 cm3. It is finite. It seems unlikely that it represents unbounded quantities for utilities.
      - JGWeissman 16 Mar 2011 22:19 UTC
        4 points
        Parent
        The Peano axioms are finite. The numbers they describe are unbounded. Finite human brains understand this.
        timtyler 16 Mar 2011 22:21 UTC
        −8 points
        Parent
        What does that have to do with how human-equivalent utility functions work?
        
        Turing machine tapes are unbounded, but real things are not—they are finite. The human brain is finite and tiny. It is not remotely unbounded.
        JGWeissman 16 Mar 2011 22:29 UTC
        0 points
        Parent
        It shows that your conclusions from human brains being finite don’t follow.
        timtyler 16 Mar 2011 23:42 UTC
        −11 points
        Parent
        No, it doesn’t. I think you folk are all barking up the wrong tree.
        
        The case for unbounded utilities rests on brains actually using something like surreal numbers to represent infinite utilities. I don’t think there’s any significant evolutionary pressure favouring such a thing—or any evidence that humans actually behave that way—but at least that is a theoretical possibility.
        
        Absent such evidence, I think Occam’s razor favours simple finite utilities that map onto the reinforcement-learning machinery evident in the brain.
        CuSithBell 17 Mar 2011 0:55 UTC
        0 points
        Parent
        
        Unbounded is not the same as infinite.
        
        timtyler 17 Mar 2011 1:10 UTC
        −1 points
        Parent
        Note that I never said or implied that it was.
        CuSithBell 17 Mar 2011 1:15 UTC
        0 points
        Parent
        You said:
        
        The case for unbounded utilities rests on brains actually… represent[ing] infinite utilities.
        
        Expand this thread
        timtyler 17 Mar 2011 1:31 UTC
        0 points
        Parent
        Exactly. That is becaues the stuff about the finite human brain represeting unboundedly huge utilities is obvious nonsense. That is why people are roping in Omega and infinite time—desperation.
      - benelliott 16 Mar 2011 22:24 UTC
        0 points
        Parent
        My 1,300 cm3 is capable of understanding the function f(x) = 3x, which is unbounded, therefore finite physical size does not prevent the brain from dealing with unbounded functions.
        
        In general, a finite machine can easily deal with unbounded numbers simply by taking unbounded amounts of time to do so. This is not as much of a problem as it may sound, since there will intevitably be an upper bound to the utilities involved in all dilemma’s I actually encounter (unless my lifespan is infinite) but not the utilities I could, in theory, compute.
        timtyler 16 Mar 2011 22:46 UTC
        −3 points
        Parent
        This is an augmented human, with a strap-on memory source bigger than the size of the planet? I thought we were probably talking about an ordinary human being—not some abstract sci-fi human that will never actually exist.
        benelliott 16 Mar 2011 23:07 UTC
        0 points
        Parent
        Who said anything about an augmented human, my comment was written in the first person except for one sentence, and I certainly don’t have a strap-on memory source bigger than a planet, but despite this I’m still pretty confident that I have an unbounded utility function.
  - Perplexed 16 Mar 2011 21:24 UTC
    0 points
    Parent
    No it is not hypothetical. If you build an AI with unbounded utility functions, yet human utility functions are (mostly) bounded, then you have built a (mostly) unfriendly AI. An AI that will be willing to sacrifice arbitrarily large amounts of current human utility in order to gain the resources to create a wonderful future for hypothetical future humans.
    - timtyler 16 Mar 2011 21:58 UTC
      −1 points
      Parent
      That’s diferent, though. The hypothetical I was objecting to was humans having unbounded utility functions. I think that idea is a case of making things up.
      
      FWIW, I stand by the idea that instrumental discounting means that debating ultimate discounting vs a lack of ultimate discounting mostly represents a storm in a teacup. In practice, all agents do instrumental discounting—since the future is uncertain and difficult to directly influence.
      
      Any debate here should really be over whether ultimate discounting on a timescale of decades is desirable—or not.
  - timtyler 17 Mar 2011 1:27 UTC
    −2 points
    Parent
    This has been rather surreal. I express what seems to me to be a perfectly ordinary position—that the finite human brain is unlikely to represent unbounded utilities—or to go in for surreal utilities—and a bunch of people have opined, that somehow, the brain does represent unboundedly large utilities—using mechanisms unspecified.
    
    When pressed, infinite quantities of time are invoked. Omega is invited onto the scene—to represent the unbounded numbers for the human. Uh...
    
    I don’t mean to be rude—but do you folk really think you are being rational here? This looks more like rationalising to me.
    
    Is there any evidence for unbounded human utilities? What would make anyone think this is so?
    - CuSithBell 17 Mar 2011 1:33 UTC
      6 points
      Parent
      Several mechanisms for expressing unbounded utility functions (NOT unbounded utilities) have been explained. The distinction has been explained. Several explicit examples have been provided.
      
      At the very least, you should update a little based on the resistance you’re experiencing.
      
      As it stands, it looks like you’re not making a good-faith attempt to understand the arguments against your position.
      - timtyler 17 Mar 2011 1:58 UTC
        0 points
        Parent
        Well, I think I can see the other side. People seem to be thinking that utility in deaths (for example) behaves linearly out to infinity. The way utilitarian ethicists dream about.
        
        I don’t think that is how the brain works. Scope insensitivity shows that most humans deal badly with the large numbers involved—in a manner quite consistent with bounded utility. There is a ceiling effect for pain and for various pleasure-inducing drugs. Those who claim to have overcome scope insensitivity haven’t really changed the underlying utility function used by the human brain. They have just tried to hack it a little—using sophisticated cultural manipulations. Their brain still uses the same finite utilities and utility functions underneath—and it can still be well-modelled that way.
        
        Indeed, I figure you will get more accurate models that way than if you project out to infinity—more accurately reproducing some types of scope insensitivity, for instance.
        CuSithBell 17 Mar 2011 3:09 UTC
        2 points
        Parent
        Sorry, I think I’m going to have to bow out at this point. It still looks like you’re arguing against fictitious positions (like “unbounded utility functions produce infinite utilities”) and failing to deal with the explicit counterexamples provided.
RobinHanson 21 Jan 2008 18:23 UTC
4 points
Carl, yes, agents who care little about the future can, if so empowered, do great damage to the future.
Peter_McCluskey2 21 Jan 2008 18:25 UTC
3 points
I agree with much of the thrust of this post. It is very bad that the causes of discount rates (such as opportunity costs) exist. But your reaction to Carl Shulman’s time travel argument leaves me wondering whether you have a coherent position. If a Friendly AI with a nonzero discount rate would conclude that it has a chance of creating time travel, and that time travel would work in a way that would abolish opportunity costs, then I would conclude that devoting a really large fraction of available resources to creating time travel is what a genuine altruist would want. Can you clarify whether you really mean to say that an AI shouldn’t devote a lot of resources toward something which would abolish opportunity costs (i.e. give everyone everything they can possibly have)? Of course, it’s not clear to me that an AI would believe it has a chance of creating time travel. And it’s not clear to me that time travel would be sufficient to abolish opportunity costs, arbitrage interest rates to zero, etc. I sometimes attempt to imagine a version of time travel which would do those things, but my mind boggles before I get close to deciding whether such a version is logically consistent. The only model of time travel I understand well enough to believe it is coherent is the one proposed by David Deutsch, which does not appear powerful enough to abolish opportunity costs or arbitrage interest rates to zero. If you were thinking of this model of time travel, then please clarify why you think it says anything interesting about the existence of discount rates.
Steven2 21 Jan 2008 18:26 UTC
2 points
There’s a typo in your math. 1/(0.95^408) = 1,226,786,652, not 12,226,786,652. But what’s a factor of 10 between friends?
steven04612 21 Jan 2008 19:30 UTC
11 points
That wasn’t me; I guess I’ll post under this name from now on.

“There is a financial argument against the possibility time travel that was published in a journal (don’t have the citation offhand): if it were possible to time-travel, interest rates would be arbitraged to zero.”

I’d say this is a special case of the Fermi Paradox for Time Travel. If people can reach us from the future, where are they?
Chris_Jeffords 21 Jan 2008 22:31 UTC
1 point
With respect to discount rates: I understand your argument(s) against the discount rate living in one’s pure preferences, but what is it you offer in its stead? No discount rate at all? Should one care the same about all time periods? Isn’t this a touch unfair for any single person who values internal discount rates? For global catastrophic risk management: should there be no discount rate applied for valuing and modeling purposes? Isn’t this the same as modeling a 0% discount rate?

With respect to AI (excuse my naivety): It seems that if a current human created AI it would ultimately bias an AI being toward having some type of human traits or incentive mapping. Otherwise we are assuming that the “human-creators” have a knowledge base beyond the understanding of “non-creator-humans” where they could create an AI which had no ties to (or resemblance of) human wants, needs, incentives, values, etc. This seems rather implausible to me.

Without omniscient human-creators, I get the feeling that an AI would be inherently biased toward having human characteristics otherwise why wouldn’t the humans creating AI try to “create” themselves in the likeness of an “ideal” AI? Furthermore, in keeping with this theme, do you think humans and an AI would have the same incentive for time travel?

Thank you for your time and consideration.
michael_vassar3 22 Jan 2008 5:53 UTC
3 points
Eliezer: Why would you assume that Pete’s utility function, or any human’s utility function is not bounded (or wouldn’t be bounded if humans had utility functions)?
Maxim_Lott 23 Jan 2008 5:07 UTC
−1 points
I think there are many serious theoretical errors in this post.

When we say that the interest rate is 5%, that means that in general people would trade $1.05 next year for $1 today. It’s basically a fact that they would be willing to do that—if people’s real discount rate were lower, they would lend money to the future at a lower interest rate. Eliezer finds it absurd that it’s 5% more important to give clean water to a family today than tomorrow, but how is it absurd when this is what consumers are saying they want for themselves as well. It’s revealed preference.

That stat and the Bruno one are also misleading because:

1) 5% is too high because it is the nominal interest rate, not the real one.

The first reason why the Giordano Bruno number is misleading is that most of that interest rate is due to inflation. Current inflation is around 3%… so that leaves about 2% of the interest rate that is due to default risk and the inconvenience of having the money tied up. Historically, inflation was probably much higher and the actual return on investment may have been closer to zero percent. It’s fine to use the nominal interest rate if we’re comparing dollars today to dollars tomorrow, but lives and clean drinking water don’t inflate like dollars do (so the interest rate on lives, so to speak, should not include that factor.)

2) “Default risk” is huge. Looking at history in retrospect is unfair.

Looking at things from the perspective of someone in Rome during 1600, a dollar could legitimately have been worth tens of thousands of times more than a promised dollar today. Rome could have been invaded in that time, the cold war could have gone nuclear, your investment company could simply have gone bankrupt or swindled you, etc. In fact, would an investment in Rome made in 1600 still be redeemable today? Would it really survive the period in Italian history labeled on wikipedia as “Foreign domination and unification (16th to 19th c.)” and Mussolini?

Any thoughts?
- soreff 22 Aug 2010 3:43 UTC
  4 points
  Parent
  
  how is it absurd when this is what consumers are saying they want for themselves as well. It’s revealed preference.
  
  Very much agreed. Perhaps one component is a kind of identity drift. I’m not quite the same person I was a year ago, nor am I quite the same person that I will be a year from now. To say that $1 I get now goes strictly to me, while $1 “I” get a year from now goes 99% to the “me” I am now and 1% to something different seems like a plausible part of the temporal preference.
tcpkac 23 Jan 2008 10:51 UTC
3 points
The answer to ‘shut up and multiply’ is ‘that’s the way people are, deal with it’. One thing apparent from these exchanges is that ‘inferential distance’ works both ways.
Tim_Freeman 29 Apr 2008 11:39 UTC
2 points
Three points in response to Eliezer’s post and one of his replies:

* A limited time horizon works better than he says. If an AI wants to put its world into a state desired by humans, and it knows that the humans don’t want to live in a galaxy that will be explode in a year, then an AI that closes its books in 1000 years will make sure that the galaxy won’t explode one year later.

* An unbounded utility works worse than he says. Recall the ^^^^ operator originally by Knuth (see http://en.wikipedia.org/wiki/Knuth%27s_up-arrow_notation) that was used in the Pascal’s Mugging article at http://lesswrong.com/lw/kd/pascals_mugging_tiny_probabilities_of_vast/.

If one allows unbounded utilities, then one has allowed a utility of about 3^^^^3 that has no low-entropy representation. In other words, there isn’t enough matter to represent a utility.

Humans have heads of a limited size that don’t use higher math to represent their desires, so bounding the utility function doesn’t limit our ability to describe human desire.

* Ad-hominem is a fallacy. The merit of a proposed FAI solution is a function of the solution, not who proposed it or how long it took them. An essential step toward overcoming bias is to train oneself not to commit well-known fallacies. There’s a good list in “The Art of Controversy” by Schopenhauer, see http://www.gutenberg.org/etext/10731.

Of course, I’m bothering to say this because I have a proposed solution out. See http://www.fungible.com/respect/paper.html.
Cosmos 23 Sep 2009 16:46 UTC
5 points
Interestingly enough, Schumpeter essentially makes this argument in his Theory of Economic Development. He is against the view that humans have intrinsic discount rates, an innate time preference, which was one of the Austrian school’s axioms. He thinks that interest is a phenomenon of economic development—resources need to be withdrawn from their customary usage, to allow entrepreneurs to find new combinations of things, and that requires compensation. Once this alternative use of resources is available, however, it becomes an opportunity cost for all other possible actions, which is the foundation of discount rates.
JGWeissman 23 Sep 2009 17:19 UTC
0 points
If an agent with no intrinsic utility discounting still has effective discounting in its instrumental values because it really can achieve exponential growth in such values, would it not still be subject to the same problem of expending all resources on attempting time travel?
- pengvado 23 Sep 2009 20:43 UTC
  2 points
  Parent
  An agent with no intrinsic utility discounting doesn’t care whether it starts an exponential investment now, or travels to a zillion years in the past and does its investing then. Either way ends up with the same total assets after any given amount of subjective time. (This assumes you can’t just chuck some money through the time machine and instantly end up with a zillion year’s interest. And that you aren’t using the time machine to avoid the end of the universe or some other time limit on future investment. And that you aren’t doing anything else interesting with it, like building a PSPACE oracle. I don’t think the original statement was about those.)
  
  Whereas an agent with intrinsic discounting would rather live in a universe where the clock says “a zillion BC” than “2009 AD”, even if the situations are otherwise identical.
XiXiDu 16 Mar 2011 18:14 UTC
2 points

[...] an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel [...]

But wouldn’t an AI without temporal discounting have an infinite incentive to expend all available resources on attempting to leave the universe to avoid the big freeze? It seems that discounting is a way to avoid Pascal’s Mugging scenarios where expected utility can outweigh tiny probabilities. Or isn’t it similar to Pascal’s Mugging if an AI tries to build a time machine regardless of the possibility of success just because the expected utility does does outweigh any uncertainty? It seems to me that in such cases one is being mugged by one’s own expectation. I suppose that is why many people disregard mere possibilities, or logical implications, if they are not backed by other kinds of evidence than their personal “betting preferences”.

Anyway, I only came across this and the Pascal’s Mugging post yesterday (which do draft my two biggest problems) only to find out that they are still unsolved problems. Or are they dissolved somewhere else?

XiXiDu’s Mugging

SIAI guy: We need money to mitigate risks from artificial intelligence.

XiXiDu: I see, but how do you know there are risks from artificial intelligence?

SIAI guy: Years worth of disjunctive lines of reasoning!

XiXiDu: Ok, so given what we know today we’ll eventually end up with superhuman AI. But we might err as we’ve been wrong in the past. Is it wise to decide against other risks in favor of risks from AI given all the uncertainty about the nature of intelligence and its possible time frame? Shouldn’t we postpone that decision?

SIAI guy: That doesn’t matter. Even given a tiny probability, the expected utility will outweigh it. If we create friendly AI we’ll save a galactic civilization from not being created. So we should err on the side of caution.
- orthonormal 15 Apr 2011 21:26 UTC
  1 point
  Parent
  This should be two separate comments, the first of which is quite insightful, but the second of which belongs in a more relevant thread.
timtyler 16 Mar 2011 20:01 UTC
−3 points

I’m writing this blog post just now, because I recently had a conversation with Carl Shulman, who proposed an argument against temporal discounting that is, as far as I know, novel: namely that an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel—maybe hunting for wormholes with a terminus in the past.

Probably not if it knows how hopeless that is—or if it has anything useful to be getting on with.

With discounting, time is of the essence—it is not to be wasted on idle fantasies.
Wei Dai 15 May 2011 22:37 UTC
21 points

And there’s worse: If your temporal discounting follows any curve other than the exponential, you’ll have time-inconsistent goals that force you to wage war against your future selves—preference reversals—cases where your self of 2008 will pay a dollar to ensure that your future self gets option A in 2011 rather than B in 2010; but then your future self in 2009 will pay another dollar to get B in 2010 rather than A in 2011.

Eliezer, you’re make non-exponential discounting out to be worse that it actually is. “Time-inconsistent goals” just means different goals, and do not “force you to wage war against your future selves” more than my having different preferences from you forces us to war against each other. One’s (non-exponential discounting) agent-moments can avoid war by conventional methods such as bargains or unilateral commitments enforced by third parties, or by more exotic methods such as application of TDT.

For your specific example, conventional game theory says that since agent_2009 moves later, backward induction implies that agent_2008 should not pay $1 since if he did, his choice would just be reversed by agent_2009. TDT-type reasoning makes this game harder to solve and seems to imply that agent_2008 might have some non-zero bargaining power, but in any case I don’t think we should expect that agent_2008 and agent_2009 each end up paying $1.
What links here?
- gwern 1 Dec 2011 0:15 UTC
  0 points
  Parent
  And of course there’s the argument that “Hyperbolic discounting is rational” given that one’s opportunities for return often bounce around a great deal.
- timtyler 18 May 2011 19:23 UTC
  0 points
  Parent
  This is often called dynamic inconsistency.
  
  It is not the end of the world—but it is easy enough to avoid.
Good_Burning_Plastic 6 Mar 2015 22:58 UTC
0 points

If you wouldn’t burn alive 1,226,786,652 people today to save Giordano Bruno from the stake in 1600

Your choice of an example makes the bullet unduly easy for me to swallow. I had to pretend you had said “to save a random peasant from pneumonia in 1600” instead for my System 1 to get your point.