And the utility function doesn’t have to be bounded by a constant. An agent will “blow out its speakers” if it follows a utility function whose dynamic range is greater than the agent’s Kolmogorov complexity + the evidence the agent has accumulated in its lifetime. The agent’s brain’s subjective probabilities will not have sufficient fidelity for such a dynamic utility function to be meaningful.
Super-exponential utility values are OK, if you’ve accumulated a super-polynomial amount of evidence.
The Solomonoff expectation value of any unbounded computable utility function diverges. This is because the program “produce the first universe with utility > n^2” is roughly of length log n + O(1) therefore it contributes 2^{-log n + O(1)} n^2 = O(n) to the expectation value.
And the utility function doesn’t have to be bounded by a constant. An agent will “blow out its speakers” if it follows a utility function whose dynamic range is greater than the agent’s Kolmogorov complexity + the evidence the agent has accumulated in its lifetime. The agent’s brain’s subjective probabilities will not have sufficient fidelity for such a dynamic utility function to be meaningful.
Super-exponential utility values are OK, if you’ve accumulated a super-polynomial amount of evidence.
The Solomonoff expectation value of any unbounded computable utility function diverges. This is because the program “produce the first universe with utility > n^2” is roughly of length log n + O(1) therefore it contributes 2^{-log n + O(1)} n^2 = O(n) to the expectation value.
Oops, that’s not quite right. But I think that something like that is right :-). 15 Quirrell points to whoever formalizes it correctly first.