I would definitely take the first of these deals, and would probably swallow the bullet and continue down the whole garden path . I would be interested to know if Eliezer’s thinking has changed on this matter since September 2009.
However, if I were building an AI which may be offered this bet for the whole human species, I would want it to use the Kelly criterion and decline, under the premise that if humans survive the next hour, there may well be bets later that could increase lifespan further. However, if the human species goes extinct at any point, then game over, we lose, the Universe is now just a mostly-cold place with a few very hot fusion fires and rocks throughout.
The Kelly criterion is, roughly, to take individual bets that maximize the expected logarithm instead of expected utility itself. Despite the VNM axioms pretty much definining utility as that-which-is-to-be-maximized, there are theorems (which I have seen, but for which I have not seen proofs yet) that the Kelly criterion is optimal in various ways. I believe, though I don’t know much about the Kelly criterion so there’s a high probability I’m wrong, that it applies to maximizing total lifespan in addition to maximizing money.
So what happens if we try to maximize log(lifespan)? The article implies we should still take the bets, but I think that’s incorrect and we wouldn’t take even one deal (except for the total freebie before the tetration garden path). To see this, note that we only care about the 80% of the worlds where we could have survived (unless Omega offers to increase this probability somewhere...), so we’ll just look at that. Now we have to choose between a 100% chance of log(life) being 10,000,000 (using base-10 log and measuring life in years), or a 99.9999% chance of log(life) being 10^10,000,000 and a 0.0001% chance of log(life) being -∞. A quick calculation shows that E(log(life)) in this case is -∞, which is far less than the E(log(life)) of 10,000,000 we get from not taking the deal.
In short, even if you accept a small chance of you dying (as you must if you want to get up in the morning), if you want the long range maximum lifespan for the human race to be as high as possible, you cannot accept even the tiniest chance of it going extinct.
This is why existential risk reduction is such a big deal; I hadn’t actually made that connection when I started writing this comment.
I would definitely take the first of these deals, and would probably swallow the bullet and continue down the whole garden path . I would be interested to know if Eliezer’s thinking has changed on this matter since September 2009.
However, if I were building an AI which may be offered this bet for the whole human species, I would want it to use the Kelly criterion and decline, under the premise that if humans survive the next hour, there may well be bets later that could increase lifespan further. However, if the human species goes extinct at any point, then game over, we lose, the Universe is now just a mostly-cold place with a few very hot fusion fires and rocks throughout.
The Kelly criterion is, roughly, to take individual bets that maximize the expected logarithm instead of expected utility itself. Despite the VNM axioms pretty much definining utility as that-which-is-to-be-maximized, there are theorems (which I have seen, but for which I have not seen proofs yet) that the Kelly criterion is optimal in various ways. I believe, though I don’t know much about the Kelly criterion so there’s a high probability I’m wrong, that it applies to maximizing total lifespan in addition to maximizing money.
So what happens if we try to maximize log(lifespan)? The article implies we should still take the bets, but I think that’s incorrect and we wouldn’t take even one deal (except for the total freebie before the tetration garden path). To see this, note that we only care about the 80% of the worlds where we could have survived (unless Omega offers to increase this probability somewhere...), so we’ll just look at that. Now we have to choose between a 100% chance of log(life) being 10,000,000 (using base-10 log and measuring life in years), or a 99.9999% chance of log(life) being 10^10,000,000 and a 0.0001% chance of log(life) being -∞. A quick calculation shows that E(log(life)) in this case is -∞, which is far less than the E(log(life)) of 10,000,000 we get from not taking the deal.
In short, even if you accept a small chance of you dying (as you must if you want to get up in the morning), if you want the long range maximum lifespan for the human race to be as high as possible, you cannot accept even the tiniest chance of it going extinct.
This is why existential risk reduction is such a big deal; I hadn’t actually made that connection when I started writing this comment.