Do you believe that a paper clip maximizer can survive in a world where another self-modifying AI exists whos value is to morph itself into the most powerful and prevalent AI in the world? I don’t see how something like a paper clip maximizer, which must split its exponential growth between becoming more powerful and creating paper clips, can ever be expected to outgrow an AI which must only become more powerful.
I realize that my statement is equivalent to saying I don’t see how FAI can ever defeat UAI. (Because FAI has more constraint on its values evolution, which must cost it something in growth rate.) So I guess I realize that the conventional wisdom here is that I am wrong, but I don’t know the reasoning that leads to my being wrong.
Yeah, if the paperclipper values a paperclip today more than a paperclip tomorrow, then I suppose it will lose out to other AIs that have a lower time discounting rate and can delay gratification for longer. Unless these other AIs also use time discounting, e.g. the power-hungry AI could value a 25% chance of ultimate power today the same as a 50% chance tomorrow.
But then again, such contests can happen only if the two AIs arise almost simultaneously. If one of them has a head start, it will try to eliminate potential competition quickly, because that’s the utility-maximizing thing to do.
I suppose that’s the main reason to be pessimistic about FAI. It’s not just that FAI is more constrained in its actions, it also takes longer to build, and a few days’ head start is enough for UAI to win.
That might be related to time discounting rates. For example, if the paperclipper has a low discounting rate (a paperclip today has the same utility as two paperclips in 100 years), and the power-hungry AI has a high discounting rate (a 25% chance of ultimate power today has the same utility as a 50% chance tomorrow), then I guess the paperclipper will tend to win. But for that contest to happen, the two AIs would need to arise almost simultaneously. If one of the AIs has a head start, it will try to takeoff quickly and stop other AIs from arising.
Do you believe that a paper clip maximizer can survive in a world where another self-modifying AI exists whos value is to morph itself into the most powerful and prevalent AI in the world? I don’t see how something like a paper clip maximizer, which must split its exponential growth between becoming more powerful and creating paper clips, can ever be expected to outgrow an AI which must only become more powerful.
I realize that my statement is equivalent to saying I don’t see how FAI can ever defeat UAI. (Because FAI has more constraint on its values evolution, which must cost it something in growth rate.) So I guess I realize that the conventional wisdom here is that I am wrong, but I don’t know the reasoning that leads to my being wrong.
Yeah, if the paperclipper values a paperclip today more than a paperclip tomorrow, then I suppose it will lose out to other AIs that have a lower time discounting rate and can delay gratification for longer. Unless these other AIs also use time discounting, e.g. the power-hungry AI could value a 25% chance of ultimate power today the same as a 50% chance tomorrow.
But then again, such contests can happen only if the two AIs arise almost simultaneously. If one of them has a head start, it will try to eliminate potential competition quickly, because that’s the utility-maximizing thing to do.
I suppose that’s the main reason to be pessimistic about FAI. It’s not just that FAI is more constrained in its actions, it also takes longer to build, and a few days’ head start is enough for UAI to win.
That might be related to time discounting rates. For example, if the paperclipper has a low discounting rate (a paperclip today has the same utility as two paperclips in 100 years), and the power-hungry AI has a high discounting rate (a 25% chance of ultimate power today has the same utility as a 50% chance tomorrow), then I guess the paperclipper will tend to win. But for that contest to happen, the two AIs would need to arise almost simultaneously. If one of the AIs has a head start, it will try to takeoff quickly and stop other AIs from arising.