Anja

• AnjaFeb 6, 2013, 9:14 PM

  4 points

  2 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: Manfred’s comment on: Save the princess: A tale of AIXI and util­ity functions

  Super hard to say without further specification of the approximation method used for the physical implementation.

• AnjaFeb 6, 2013, 9:11 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: Save the princess: A tale of AIXI and util­ity functions

  So I would only consider the formulation in terms of semimeasures to be satisfactory if the semimeasures are specific enough that the correct semimeasure plus the observation sequence is enough information to determine everything that’s happening in the environment.

  Can you make an example of a situation in which that would not be the case? I think the semimeasure AIXI and deterministic programs AIXI are pretty much equivalent, am I overlooking something here?

  If we’re going to allow infinite episodic utilities, we’ll need some way of comparing how big different nonconvergent series are.

  I think we need that even without infinite episodic utilities. I still think there might be possibilities involving surreal numbers, but I haven’t found the time yet to develop this idea further.

  Why?

  Because otherwise we definitely end up with an unenumerable utility function and every approximation will be blind between infinitely many futures with infinitely large utility differences, I think. The set of all binary strings of infinite length is uncountable and how would we feed that into an enumerable/​computable function? Your approach avoids that via the use of policies p and q that are by definition computable.

• AnjaFeb 6, 2013, 8:57 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: Squark’s comment on: Save the princess: A tale of AIXI and util­ity functions

  I think you are proposing to have some hypotheses privileged in the beginning of Solomonoff induction, but not too much because the uncertainty helps fight wireheading by means of providing knowledge about the existence of an idealized, “true” utility function and world model. I that a correct summary? (Just trying to test whether I understand what you mean.)

  In particular they can make positive use of wire-heading to reprogram themselves even if the basic architecture M doesn’t allow it

  Can you explain this more?

Save the princess: A tale of AIXI and util­ity functions

• AnjaJan 5, 2013, 2:01 PM

  3 points

  2 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: In­ter­per­sonal and in­trap­er­sonal util­ity comparisons

  They just do interpersonal comparisons; lots of their ideas generalize to intrapersonal comparisons though.

• AnjaJan 4, 2013, 2:35 PM

  2 points

  2 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  on: In­ter­per­sonal and in­trap­er­sonal util­ity comparisons

  I recommend the book “Fair Division and Collective Welfare” by H. J. Moulin, it discusses some of these problems and several related others.

• AnjaDec 20, 2012, 9:36 PM

  2 points

  2 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: A util­ity-max­i­miz­ing vari­ent of AIXI

  True. :)

• AnjaDec 20, 2012, 9:35 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: A util­ity-max­i­miz­ing vari­ent of AIXI

  I get that now, thanks.

• AnjaDec 20, 2012, 6:29 PM

  2 points

  2 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: A util­ity-max­i­miz­ing vari­ent of AIXI

  you forgot to multiply by 2^-l(q)

  I think then you would count that twice, wouldn’t you? Because my original formula already contains the Solomonoff probability...

• AnjaDec 20, 2012, 6:25 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: AlexMennen’s comment on: A util­ity-max­i­miz­ing vari­ent of AIXI

  Let’s stick with delusion boxes for now, because assuming that we can read off from the environment whether the agent has wireheaded breaks dualism. So even if we specify utility directly over environments, we still need to master the task of specifying which action/​environment combinations contain delusion boxes to evaluate them correctly. It is still the same problem, just phrased differently.

• AnjaDec 19, 2012, 6:08 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  on: A util­ity-max­i­miz­ing vari­ent of AIXI

  I think there is something off with the formulas that use policies: If you already choose the policy

  

  =y_{%3Ck}y_k)

  then you cannot choose an y_k in the argmax.

  Also for the Solomonoff prior you must sum over all programs

  

  =x_{1:m_k}) .

  Could you maybe expand on the proof of Lemma 1 a little bit? I am not sure I get what you mean yet.

• AnjaDec 19, 2012, 5:41 PM

  6 points

  5 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  on: A util­ity-max­i­miz­ing vari­ent of AIXI

  I like how you specify utility directly over programs, it describes very neatly how someone who sat down and wrote a utility function

  

  )

  would do it: First determine how the observation could have been computed by the environment and then evaluate that situation. This is a special case of the framework I wrote down in the cited article; you can always set

  

  =\sum_{q:q(y_{1:m_k})=x_{1:m_k}}%20U(q,y_{1:m_k}))

  This solves wireheading only if we can specify which environments contain wireheaded (non-dualistic) agents, delusion boxes, etc..

• AnjaNov 29, 2012, 11:36 PM

  0 points

  0 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: brazil84’s comment on: A defi­ni­tion of wireheading

  You are a wirehead if you consider your true utility function to be genetic fitness.

• AnjaNov 29, 2012, 3:41 AM

  6 points

  4 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: davidpearce’s comment on: A defi­ni­tion of wireheading

  To what extent does our response to Nozick’s Experience Machine Argument typically reflect status quo bias rather than a desire to connect with ultimate reality?

  I think the argument that people don’t really want to stay in touch with reality but rather want to stay in touch with their past makes a lot of sense. After all we construct our model of reality from our past experiences. One could argue that this is another example of a substitute measure, used to save computational resources: Instead of caring about reality we care about our memories making sense and being meaningful.

  On the other hand I assume I wasn’t the only one mentally applauding Neo for swallowing the red pill.

• AnjaNov 29, 2012, 3:27 AM

  1 point

  1 vote

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: Alexei’s comment on: A defi­ni­tion of wireheading

  Thank you.

• AnjaNov 29, 2012, 3:27 AM

  5 points

  4 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: devas’s comment on: A defi­ni­tion of wireheading

  What would happen if we set an algorithm inside the AGI assigning negative infinite utility to any action which modifies its own utility function and said algorithm itself?

  There are several problems with this approach: First of all how do you specify all actions that modify the utility function? How likely do you think it is that you can exhaustively specify all sequences of actions that lead to modification of the utility function in a practical implementation? Experience with cryptography has taught us, that there is almost always some side channel attack that the original developers have not thought of, and that is just in the case of human vs. human intelligence.

  Forbidden actions in general seem like a bad idea with an AGI that is smarter than us, see for example the AI Box experiment.

  Then there is the problem that we actually don’t want any part of the AGI to be unmodifiable. The agent might revise its model of how the universe works (like we did when we went from Newtonian physics to quantum mechanics) and then it has to modify its utility function or it is left with gibberish.

  All that said, I think what you described corresponds to the hack evolution has used on us: We have acquired a list of things (or schemas) that will mess up our utility functions and reduce agency and those just feel icky to us, like the experience machine or electrical stimulation of the brain. But we don’t have the luxury of learning by making lots and lots of mistakes that evolution had.

• AnjaNov 29, 2012, 2:54 AM

  4 points

  3 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: timtyler’s comment on: A defi­ni­tion of wireheading

  You might be right. I thought about this too, but it seemed people on LW had already categorized the experience machine as wireheading. If we rebrand, we should maybe say “self-delusion” instead of “pornography problem”; I really like the term “utility counterfeiting” though and the example about counterfeit money in your essay.

• AnjaNov 28, 2012, 1:51 AM

  13 points

  8 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: Eliezer Yudkowsky’s comment on: A defi­ni­tion of wireheading

  The word “value” seems unnecessarily value-laden here.

  Changed it to “number”.

• AnjaNov 27, 2012, 11:37 PM

  8 points

  5 votes

  Overall karma indicates overall quality.

  0

  0 votes

  Agreement karma indicates agreement, separate from overall quality.

  in reply to: dspeyer’s comment on: A defi­ni­tion of wireheading

  You are correct in pointing out that for human agents the evaluation procedure is not a deliberate calculation of expected utility, but some messy computation we have little access to. In many instances this can however be reasonably well translated into the framework of (partial) utility functions, especially if our preferences approximately satisfy transitivity, continuity and independence.

  For noticing discrepancies between true and substitute utility it is not necessary to exactly know both functions, it suffices to have an icky feeling that tells you that you are acting in a way that is detrimental to your (true) goals.

  If all else fails we can time-index world states and equip the agent with a utility function by pretending that he assigned utility of 1 to the world state he actually brought about and 0 to the others. ;)

A defi­ni­tion of wireheading