Ok, I might have formulated myself badly. My argument is that any agent of bounded computational power is forced to use two utility functions. The one they wish they had (i.e. the unbounded linear version) and the one they are forced to use in their calculations because of their limitations (i.e. an asymptotically bounded approximation).
For those agents capable of self-modification, just add a clause to increase their computational power (and thereby increasing the bound of their approximation) whenever the utilities of the “scales they’re working on” differ by more than some small specified number.
So, my answer to this person would be “stick around until I can safely modify myself into dealing with your request”, or alternatively, if he wants an answer right now after seeing his evidence, “here’s 5 dollars”.
Ok, I might have formulated myself badly. My argument is that any agent of bounded computational power is forced to use two utility functions. The one they wish they had (i.e. the unbounded linear version) and the one they are forced to use in their calculations because of their limitations (i.e. an asymptotically bounded approximation).
For those agents capable of self-modification, just add a clause to increase their computational power (and thereby increasing the bound of their approximation) whenever the utilities of the “scales they’re working on” differ by more than some small specified number.
So, my answer to this person would be “stick around until I can safely modify myself into dealing with your request”, or alternatively, if he wants an answer right now after seeing his evidence, “here’s 5 dollars”.