Notice that Eliezer’s definition + the universal prior is essentially the same as the computationally unbounded variant of my definition of goal-directed intelligence, if you fix the utility function and prior. I think that when you’re comparing different utility functions in your definition of strategy-stealing, it’s better to emulate my definition and correct by the description complexity of the utility function. Indeed, if you have an algorithm that works equally well for all utility functions but contains a specification of the utility function inside its source code, then you get “spurious” variance if you don’t correct for it (usually superior algorithms would also have to contain a specification of the utility function, so the more complex the utility function the less of them will be).
I think that when you’re comparing different utility functions in your definition of strategy-stealing, it’s better to emulate my definition and correct by the description complexity of the utility function.
Yep, that’s a good point—I agree that correcting for the description length of the utility function is a good idea there.
Just to check, the correction you both have in mind is to add the complexity of the utility to the optimization powers before taking the standard deviation?
Indeed, if you have an algorithm that works equally well for all utility functions but contains a specification of the utility function inside its source code, then you get “spurious” variance if you don’t correct for it
Is the issue in that case that the hardcoded utility will require far less optimization power?
If it’s that, I’m not sure why this is considered a problem. After all, such an algorithm is not really compatible with strategy stealing, even if it could become so quite easily. This seems like a distinction we should ask from the definition.
Notice that Eliezer’s definition + the universal prior is essentially the same as the computationally unbounded variant of my definition of goal-directed intelligence, if you fix the utility function and prior. I think that when you’re comparing different utility functions in your definition of strategy-stealing, it’s better to emulate my definition and correct by the description complexity of the utility function. Indeed, if you have an algorithm that works equally well for all utility functions but contains a specification of the utility function inside its source code, then you get “spurious” variance if you don’t correct for it (usually superior algorithms would also have to contain a specification of the utility function, so the more complex the utility function the less of them will be).
Yep, that’s a good point—I agree that correcting for the description length of the utility function is a good idea there.
Just to check, the correction you both have in mind is to add the complexity of the utility to the optimization powers before taking the standard deviation?
Is the issue in that case that the hardcoded utility will require far less optimization power?
If it’s that, I’m not sure why this is considered a problem. After all, such an algorithm is not really compatible with strategy stealing, even if it could become so quite easily. This seems like a distinction we should ask from the definition.