Can we formalize AIXI and the intelligence measure in terms of utility functions, instead?
No. Preference utility functions are defined on world states, which are, in the general case, not completely accessible to the agent.
AIXI discounts the future by requiring that total future reward is bounded, and therefore so does the intelligence measure. This seems to me like a constraint that does not reflect reality
If cumulative reward is not bounded, you might end up comparing infinities. Anyway, I suppose they chose to have a fixed horizon rather than, say, exponential discounting, so that at any prediction step any individual predictor program needs to run only for finite time. Otherwise AIXI would be, in some sense, doubly uncomputable.
Perhaps by using an average of utility functions weighted by their K-complexity
That yields worst-case complexity.
This might okay if your agent is a playing arbitrary games on a computer, but if you are trying to determine how powerful an agent will be in this universe, you probably want to replace the Solomonoff prior with the posterior resulting from updating the Solomonoff prior with data from our universe.
That’s what the agent is supposed to do by itself.
No. Preference utility functions are defined on world states, which are, in the general case, not completely accessible to the agent.
If cumulative reward is not bounded, you might end up comparing infinities. Anyway, I suppose they chose to have a fixed horizon rather than, say, exponential discounting, so that at any prediction step any individual predictor program needs to run only for finite time. Otherwise AIXI would be, in some sense, doubly uncomputable.
That yields worst-case complexity.
That’s what the agent is supposed to do by itself.