Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.
Can’t this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.
Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)
The simple algorithm is the greedy decision-finding method “Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions”, which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).
But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.
But even setting those aside: this isn’t any more of a solution than the version without discounting. They’re similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.
Can’t this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.
Finding a problem with the simple algorithm that usually gives you a good outcome doesn’t mean you get to choose a new utility function.
Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)
The simple algorithm is the greedy decision-finding method “Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions”, which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).
But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.
But even setting those aside: this isn’t any more of a solution than the version without discounting. They’re similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.
Yes, but Eliezer doesn’t believe in discounting terminal values.
So, let’s be clear—are we talking about what works, or what we think Eliezer is dumb for believing?
Well, first I’m not a consequentialist.
However, the linked post has a point, why should we value future live less?