[...] an AI with a 5% temporal discount rate has a nearly infinite incentive to expend all available resources on attempting time travel [...]
But wouldn’t an AI without temporal discounting have an infinite incentive to expend all available resources on attempting to leave the universe to avoid the big freeze? It seems that discounting is a way to avoid Pascal’s Mugging scenarios where expected utility can outweigh tiny probabilities. Or isn’t it similar to Pascal’s Mugging if an AI tries to build a time machine regardless of the possibility of success just because the expected utility does does outweigh any uncertainty? It seems to me that in such cases one is being mugged by one’s own expectation. I suppose that is why many people disregard mere possibilities, or logical implications, if they are not backed by other kinds of evidence than their personal “betting preferences”.
Anyway, I only came across this and the Pascal’s Mugging post yesterday (which do draft my two biggest problems) only to find out that they are still unsolved problems. Or are they dissolved somewhere else?
XiXiDu’s Mugging
SIAI guy: We need money to mitigate risks from artificial intelligence.
XiXiDu: I see, but how do you know there are risks from artificial intelligence?
SIAI guy: Years worth of disjunctive lines of reasoning!
XiXiDu: Ok, so given what we know today we’ll eventually end up with superhuman AI. But we might err as we’ve been wrong in the past. Is it wise to decide against other risks in favor of risks from AI given all the uncertainty about the nature of intelligence and its possible time frame? Shouldn’t we postpone that decision?
SIAI guy: That doesn’t matter. Even given a tiny probability, the expected utility will outweigh it. If we create friendly AI we’ll save a galactic civilization from not being created. So we should err on the side of caution.
But wouldn’t an AI without temporal discounting have an infinite incentive to expend all available resources on attempting to leave the universe to avoid the big freeze? It seems that discounting is a way to avoid Pascal’s Mugging scenarios where expected utility can outweigh tiny probabilities. Or isn’t it similar to Pascal’s Mugging if an AI tries to build a time machine regardless of the possibility of success just because the expected utility does does outweigh any uncertainty? It seems to me that in such cases one is being mugged by one’s own expectation. I suppose that is why many people disregard mere possibilities, or logical implications, if they are not backed by other kinds of evidence than their personal “betting preferences”.
Anyway, I only came across this and the Pascal’s Mugging post yesterday (which do draft my two biggest problems) only to find out that they are still unsolved problems. Or are they dissolved somewhere else?
XiXiDu’s Mugging
SIAI guy: We need money to mitigate risks from artificial intelligence.
XiXiDu: I see, but how do you know there are risks from artificial intelligence?
SIAI guy: Years worth of disjunctive lines of reasoning!
XiXiDu: Ok, so given what we know today we’ll eventually end up with superhuman AI. But we might err as we’ve been wrong in the past. Is it wise to decide against other risks in favor of risks from AI given all the uncertainty about the nature of intelligence and its possible time frame? Shouldn’t we postpone that decision?
SIAI guy: That doesn’t matter. Even given a tiny probability, the expected utility will outweigh it. If we create friendly AI we’ll save a galactic civilization from not being created. So we should err on the side of caution.
This should be two separate comments, the first of which is quite insightful, but the second of which belongs in a more relevant thread.