I am quite sure that pareto optimality is untouched by the proposed changes, but I haven’t written down a proof yet.
Anja
Karma: 235
Universal agents and utility functions
I am pretty sure that Agent 2 will wirehead on the Simpleton Gambit, depending heavily on the number of time cycles to follow, the comparative advantage that can be gained from wireheading and the negative utility the current utility function assigns to the change.
Agent 1 will have trouble modeling how its decision to change its utility function now will influence its own decisions later, as described in AIXI and existential despair. So basically the two futures look very similar to the agent except that for the part where the screen says something different and then it all comes down to whether the utility function has preferences over that particular fact.
I would assume that it is not smart enough to forsee its own future actions and therefore dynamically inconsistent. The original AIXI does not allow for the agent to be part of the environment. If we tried to relax the dualism then your question depends strongly on the approximation to AIXI we would use to make it computable. If this approximation can be scaled down in a way such that it is still a good estimator for the agent’s future actions, then maybe an environment containing a scaled down, more abstract AIXI model will, after a lot of observations, become one of the consistent programs with lowest complexity. Maybe. That is about the only way I can imagine right now that we would not run into this problem.
This generalizes to the horizon problem: If at time k you only look ahead to time step m_k but have unlimited life span you will make infinitely large mistakes.
First, replace the action-perception sequence with an action-perception-utility sequence u1,y1,x1,u2,y2,x2,etc.
This seems unnecessary. The information u_i is already contained in x_i.
modeled_action(n, k) = argmax(y_k) uk(yx\<k, yx_k:n)*M(uyx_<k, uyx_k:n)
This completely breaks the expectimax principle. I assume you actually mean something like
=\textrm{arg}\max_{y_k}\sum_{x_k}u_k(\.{y}\.{x}_{%3Ck}y\underline{x}_{k:n})M(\.{y}\.{x}_{%3Ck}y\underline{x}_{k:n}))which is just Agent 2 in disguise.
I second the general sentiment that it would be good for an agent to have these traits, but if I follow your equations I end up with Agent 2.
I am starting to see what you mean. Let’s stick with utility functions over histories of length m_k (whole sequences) like you proposed and denote them with a capital U to distinguish them from the prefix utilities. I think your Agent 4 runs into the following problem: modeled_action(n,m) actually depends on the actions and observations yx_{k:m-1} and needs to be calculated for each combination, so y_m is actually
)which clutters up the notation so much that I don’t want to write it down anymore.
We also get into trouble with taking the expectation, the observations x_{k+1:n} are only considered in modeling the actions of the future agents, but not now. What is M(yx_<k,yx_k:n) even supposed to mean, where do the x’s come from?
So let’s torture some indices:
=\textrm{arg}\max_{y_n}\sum_{x_{n:m_k}}U_n(yx_{1:n}\hat{y}_{n+1,k}(yx_{1:n})x_{n+1}\dots)x_{m_k})M(\.{y}\.{x}_{%3Ck}yx_{k:n-1}\hat{y}\underline{x}_{n:m_k}))where n>=k and
This is not really AIXI anymore and I am not sure what to do with it, but I like it.
There is also a more detailed paper by Lattimore and Hutter (2011) on discounting and time consistency that is interesting in that context.
A definition of wireheading
You are correct in pointing out that for human agents the evaluation procedure is not a deliberate calculation of expected utility, but some messy computation we have little access to. In many instances this can however be reasonably well translated into the framework of (partial) utility functions, especially if our preferences approximately satisfy transitivity, continuity and independence.
For noticing discrepancies between true and substitute utility it is not necessary to exactly know both functions, it suffices to have an icky feeling that tells you that you are acting in a way that is detrimental to your (true) goals.
If all else fails we can time-index world states and equip the agent with a utility function by pretending that he assigned utility of 1 to the world state he actually brought about and 0 to the others. ;)
The word “value” seems unnecessarily value-laden here.
Changed it to “number”.
You might be right. I thought about this too, but it seemed people on LW had already categorized the experience machine as wireheading. If we rebrand, we should maybe say “self-delusion” instead of “pornography problem”; I really like the term “utility counterfeiting” though and the example about counterfeit money in your essay.
What would happen if we set an algorithm inside the AGI assigning negative infinite utility to any action which modifies its own utility function and said algorithm itself?
There are several problems with this approach: First of all how do you specify all actions that modify the utility function? How likely do you think it is that you can exhaustively specify all sequences of actions that lead to modification of the utility function in a practical implementation? Experience with cryptography has taught us, that there is almost always some side channel attack that the original developers have not thought of, and that is just in the case of human vs. human intelligence.
Forbidden actions in general seem like a bad idea with an AGI that is smarter than us, see for example the AI Box experiment.
Then there is the problem that we actually don’t want any part of the AGI to be unmodifiable. The agent might revise its model of how the universe works (like we did when we went from Newtonian physics to quantum mechanics) and then it has to modify its utility function or it is left with gibberish.
All that said, I think what you described corresponds to the hack evolution has used on us: We have acquired a list of things (or schemas) that will mess up our utility functions and reduce agency and those just feel icky to us, like the experience machine or electrical stimulation of the brain. But we don’t have the luxury of learning by making lots and lots of mistakes that evolution had.
Thank you.
To what extent does our response to Nozick’s Experience Machine Argument typically reflect status quo bias rather than a desire to connect with ultimate reality?
I think the argument that people don’t really want to stay in touch with reality but rather want to stay in touch with their past makes a lot of sense. After all we construct our model of reality from our past experiences. One could argue that this is another example of a substitute measure, used to save computational resources: Instead of caring about reality we care about our memories making sense and being meaningful.
On the other hand I assume I wasn’t the only one mentally applauding Neo for swallowing the red pill.
You are a wirehead if you consider your true utility function to be genetic fitness.
I like how you specify utility directly over programs, it describes very neatly how someone who sat down and wrote a utility function
)would do it: First determine how the observation could have been computed by the environment and then evaluate that situation. This is a special case of the framework I wrote down in the cited article; you can always set
=\sum_{q:q(y_{1:m_k})=x_{1:m_k}}%20U(q,y_{1:m_k}))This solves wireheading only if we can specify which environments contain wireheaded (non-dualistic) agents, delusion boxes, etc..
I think there is something off with the formulas that use policies: If you already choose the policy
=y_{%3Ck}y_k)then you cannot choose an y_k in the argmax.
Also for the Solomonoff prior you must sum over all programs
=x_{1:m_k}) .Could you maybe expand on the proof of Lemma 1 a little bit? I am not sure I get what you mean yet.
Took the survey. Does the god question include simulators? I answered under the assumption that it did not.