Given how many times Eliezer has linked to it, it’s a little surprising that nobody seems to have picked up on this yet, but the paragraph about the utility function not being up for grabs seems to have a pretty serious technical flaw:
There is no finite amount of life lived N where I would prefer a 80.0001% probability of living N years to an 0.0001% chance of living a googolplex years and an 80% chance of living forever. This is a sufficient condition to imply that my utility function is unbounded.
Let p = 80% and let q be one in a million. I’m pretty sure that what Eliezer has in mind is,
(A) For all n, there is an even larger n’ such that (p+q)u(live n years) < pu(live n’ years) + q*(live a googolplex years).
This indeed means that {u(live n’ years) | n’ in N} is not upwards bounded—I did check the math :-) --, which means that u is not upwards bounded, which means that u is not bounded. But what he actually said was,
(B) For all n, (p+q)u(live n years) ⇐ pu(live forever) + q*u(live googolplex years)
That’s not only different from A, it contradicts A! It doesn’t imply that u needs to be bounded, of course, but it flat out states that {u(live n years) | n in N} is upwards bounded by (pu(live forever) + qu(live googolplex years))/(p+q).
(We may perhaps see this as reason enough to extend the domain of our utility function to some superset of the real numbers. In that case it’s no longer necessary for the utility function to be unbounded to satisfy (A), though—although we might invent a new condition like “not bounded by a real number.”)
Given how many times Eliezer has linked to it, it’s a little surprising that nobody seems to have picked up on this yet, but the paragraph about the utility function not being up for grabs seems to have a pretty serious technical flaw:
Let p = 80% and let q be one in a million. I’m pretty sure that what Eliezer has in mind is,
(A) For all n, there is an even larger n’ such that (p+q)u(live n years) < pu(live n’ years) + q*(live a googolplex years).
This indeed means that {u(live n’ years) | n’ in N} is not upwards bounded—I did check the math :-) --, which means that u is not upwards bounded, which means that u is not bounded. But what he actually said was,
(B) For all n, (p+q)u(live n years) ⇐ pu(live forever) + q*u(live googolplex years)
That’s not only different from A, it contradicts A! It doesn’t imply that u needs to be bounded, of course, but it flat out states that {u(live n years) | n in N} is upwards bounded by (pu(live forever) + qu(live googolplex years))/(p+q).
(We may perhaps see this as reason enough to extend the domain of our utility function to some superset of the real numbers. In that case it’s no longer necessary for the utility function to be unbounded to satisfy (A), though—although we might invent a new condition like “not bounded by a real number.”)