One problem is that with UFAI, we are not interested (I assume) in optimization power with respect to a goal other than UFAI’s own (UFAI doesn’t optimize human value very well). Since rigorous elicitation of agent’s own goal is one of the great unsolved problems in FAI theory, a definition of optimization power might be somewhat FAI-theory-complete...
Since rigorous elicitation of agent’s own goal is one of the great unsolved problems in FAI theory, a definition of optimization power might be somewhat FAI-theory-complete...
I would suggest GAI-theory-complete rather than FAI-theory-complete. FAI requires several additional considerations that go beyond that required for analysis of or implementation of an optimization algorithm with a goal system.
Could you evaluate against all possible goals and see which scores the highest in terms of optimization power?
Immediate problems with my suggestion:
It’s not very practical. There are a lot of possible functions to consider.
It would tend to lead to the conclusion that everyone’s goal is to ensure the laws of physics are being followed, and that they’re succeeding perfectly.
One problem is that with UFAI, we are not interested (I assume) in optimization power with respect to a goal other than UFAI’s own (UFAI doesn’t optimize human value very well). Since rigorous elicitation of agent’s own goal is one of the great unsolved problems in FAI theory, a definition of optimization power might be somewhat FAI-theory-complete...
I would suggest GAI-theory-complete rather than FAI-theory-complete. FAI requires several additional considerations that go beyond that required for analysis of or implementation of an optimization algorithm with a goal system.
Could you evaluate against all possible goals and see which scores the highest in terms of optimization power?
Immediate problems with my suggestion:
It’s not very practical. There are a lot of possible functions to consider.
It would tend to lead to the conclusion that everyone’s goal is to ensure the laws of physics are being followed, and that they’re succeeding perfectly.