recursive self-improvement demands goal-stability (P5), otherwise a rational agent wouldn’t self-improve, as it would not be instrumental to its current goals. For example, if a paperclip maximizer wouldn’t be able to tell that once after it improves its intelligence dramatically, it would still be paperclip-friendly, it wouldn’t risk self-improvement until it was able to prove paperclip-friendliness.
It “wouldn’t risk it?” And yet one might think there’s some reward that someone would be willing to take a chance for—if currently you generate 10 utility per day and you have a chance to increase that to 100, you should do it if you have a better than 1⁄10 chance (if the other 9⁄10 are 0 utility per day).
Further, to compute a cost-benefit analysis of self-improvement and to measure its success, an AGI will need highly specific goal-parameters (P6), i.e. a well-defined utility-function.
The AI could have any decision-choosing system it wants. It could calculate utilities precisely and compare that with a thorough utility function, or on the other hand it could have a list of a few thousand rules it followed as best it could, weighting rules by a time-inconsistent method like priming. If the question is “are there practical (though not necessarily safe) decision-choosing systems other than utility?” I’d say the answer is yes.
It “wouldn’t risk it?” And yet one might think there’s some reward that someone would be willing to take a chance for—if currently you generate 10 utility per day and you have a chance to increase that to 100, you should do it if you have a better than 1⁄10 chance (if the other 9⁄10 are 0 utility per day).
The AI could have any decision-choosing system it wants. It could calculate utilities precisely and compare that with a thorough utility function, or on the other hand it could have a list of a few thousand rules it followed as best it could, weighting rules by a time-inconsistent method like priming. If the question is “are there practical (though not necessarily safe) decision-choosing systems other than utility?” I’d say the answer is yes.