Karl

• Karl 24 Oct 2014 21:12 UTC

  5 points

  in reply to: lackofcheese’s comment on: In­tro­duc­ing Cor­rigi­bil­ity (an FAI re­search sub­field)

  Firstly, the important part of my modification to the indifference formalism is not about conditioning on the actual o but it’s the fact that in evaluating the expectation of UN it take the action in A2 (for a given pair (a1,o)) which maximize UN instead of the action which maximize U (note that U is equal to US in the case that o is not in Press.).

  Secondly an agent which chose a1 by simply maximizing E[UN | NotPress; a1] + E[US | Press; a1] do exhibit pathological behaviors. In partcular, there will still be incentives to manage the news, but from both sides now (there is an incentive to cause the button to be pressed in the event of an information which is bad news from the point of view of UN and incentives to cause the button to not be pressed in the events of information which is bad news from the point of view of US.

• Karl 24 Oct 2014 4:00 UTC

  3 points

  in reply to: interstice’s comment on: In­tro­duc­ing Cor­rigi­bil­ity (an FAI re­search sub­field)

  In the indifference formalism the agent in selecting A1 act like a UN agent that believe that the shutdown button will not be pressed, therefore it create perverse incentives to “manage the news”. Which means that if the agent can cause his shutdown button to be pressed in the event of bad news, it will.

  My formulation avoid this pathological behavior by instead making the agent select A1 as if it was a UN-agent which believed that it would continue to optimize according to UN even in the vent of the button being pressed which avoid the perverse incentives to “manage the news”, while still not having any incentives to avoid the button being pressed because the agent will act like it believe that pressing the button will not cause it to initiate a shutdown.

• Karl 24 Oct 2014 2:38 UTC

  6 points

  on: In­tro­duc­ing Cor­rigi­bil­ity (an FAI re­search sub­field)

  Why not make it so that the agent in selecting A1 act as a UN-agent that believe that it will continue to optimize according to UN even in the event of the button being pressed rather than a UN agent that believe that the button will never be pressed: that is pick U such that

  U(a1,o,a2) = UN(a1,o,a2) if o is in Press or US(a1,o,a2) + f(a1,o) - g(a1,o) if o is not in Press

  where f(a1,o) is the maximum value of UN(a1,o,b) for b in A2 and g(a1,o) is the maximum value of US(a1,o,b) for b in A2.

  This would avoid the perverse manipulation incentives problem detailed on section 4.2 of the paper.

• Karl 11 Nov 2013 19:23 UTC

  2 points

  on: Re­duced im­pact AI: no back channels

  Apart from the obvious problems with this approach, (The AI can do a lot with the output channel other than what you wanted it to do, choosing an appropriate value for λ, etc.) I don’t see why this approach would be any easier to implement than CEV.

  Once you know what a bounded approximation of an ideal algorithm is supposed to look like, how the bounded version is supposed to reason about it’s idealised version and how to refer to arbitrary physical data, as the algorithm defined in your post assume, then implementing CEV really doesn’t seem to be that hard of a problem.

  So could you explain why you believe that implementing CEV would be so much harder than what you propose in your post?

• Karl 24 Jul 2013 23:39 UTC

  4 points

  in reply to: cousin_it’s comment on: An ar­gu­ment against in­di­rect normativity

  By that term I simply mean Eliezer’s idea that the correct decision theory ought to use a maximization vantage points with a no-blackmail equilibrium.

• Karl 24 Jul 2013 23:14 UTC

  0 points

  on: An ar­gu­ment against in­di­rect normativity

  If the NBC (No Blackmail Conjecture) is correct, then that shouldn’t be a problem.

• Karl 24 Jul 2013 14:37 UTC

  1 point

  in reply to: linkhyrule5’s comment on: The Empty White Room: Sur­real Utilities

  So you wouldn’t trade whatever amount of time Frank as left, which is at most measured in decades, against a literal eternity of Fun?

  If I was Frank in this scenario, I would tell the other guy to accept the deal.

• Karl 16 Jul 2013 16:48 UTC

  4 points

  in reply to: [deleted]’s comment on: An At­tempt at Prefer­ence Uncer­tainty Us­ing VNM

  Hmm. I’ll have to take a closer look at that. You mean that the uncertainties are correlated, right?

  No. To quote your own post:

  A similar process allows us to arbitrarily set exactly one of the km.

  I meant that the utility function resulting from averaging over your uncertainty over the km’s will depend on which km you chose to arbitrarily set in this way. I gave an example of this phenomenon in my original comment.

• Karl 16 Jul 2013 15:02 UTC

  6 points

  on: An At­tempt at Prefer­ence Uncer­tainty Us­ing VNM

  You can’t simply average the km’s. Suppose you estimate .5 probability that k2 should be twice k1 and .5 probability that k1 should be twice k2. Then if you normalize k1 to 1, k2 will average to 1.25, while similarly if you normalize k2 to 1, k1 will average to 1.25.

  In general, to each choice of km’s will correspond a utility function and the utility function we should use will be a linear combination of those utility functions and we will have renormalization parameters k’m and, if we accept the argument given in your post, those k’m ought to be just as dependant on your preferences, so you’re probably also uncertain about the values that those parameters should take and so you obtain k″m’s and so on ad infinitum. So you obtain an infinite tower of uncertain parameters and it isn’t obvious how to obtain a utility function out of this mess.

• Karl 15 Jul 2013 3:24 UTC

  3 points

  in reply to: fractalman’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  What do you even mean by “is a possible outcome” here? Do you mean that there is no proof in PA of the negation of the proposition?

  The formula of a modal agent must be fully modalized, which means that all propositions containing references to actions of agents within the formula must be within the scope of a provability operator.

• Karl 15 Jul 2013 3:13 UTC

  5 points

  in reply to: AlexMennen’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  Proof without using Kripke semantic: Let X be a modal agent and Phi(...) it’s associated fully modalized formula. Then if PA was inconsistent Phi(...) would reduce to a truth value independent of X opponent and so X would play the same move against both FairBot and UnfairBot (and this is provable in PA). But PA cannot prove it’s own consistency so PA cannot both prove X(FairBot) = C and X(UnfairBot) = D and so we can’t both have FairBot(X) = C and UnfairBot(X) = C. QED

• Karl 14 Jul 2013 13:18 UTC

  2 points

  in reply to: AlexMennen’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  Proof: Let X be a modal agent, Phi(...) it’s associated fully modalized formula, (K, R) a GL Kripke model and w minimal in K. Then, for all statement of the form ◻(...) we have w |- ◻(...) so Phi(...) reduce in w to a truth value which is independent of X opponent. As a result, we can’t have both w |- X(FairBot) = C and w |- X(UnfairBot) = D and so we can’t have both ◻(X(FairBot) = C) and ◻(X(UnfairBot) = D) and so we can’t both have FairBot(X) = C and UnfairBot(X) = C. QED

• Karl 14 Jul 2013 2:53 UTC

  2 points

  in reply to: fractalman’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  no modal agent can get both FairBot and UnfairBot to cooperate with it.

  TrollDetector is not a modal agent.

• Karl 14 Jul 2013 0:18 UTC

  4 points

  in reply to: AlexMennen’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  UnfairBot defect against PrudentBot.

  Proof: For UnfairBot to cooperate with PrudentBot, PA would have to prove that PrudentBot defect against UnfairBot which would require PA to prove that “PA does not prove that UnfairBot cooperate with PrudentBot or PA+1 does not prove that UnfairBot defect against DefectBot” but that would require PA to prove it’s own consistency which it cannot do. QED

• Karl 10 Jul 2013 22:08 UTC

  4 points

  on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  Here is another obstacle to an optimality result: define UnfairBot as the agent which cooperate with X if and only if PA prove that X defect against UnfairBot, then no modal agent can get both FairBot and UnfairBot to cooperate with it.

• Karl 11 Jun 2013 2:09 UTC

  2 points

  in reply to: Will_Sawin’s comment on: Ro­bust Co­op­er­a­tion in the Pri­soner’s Dilemma

  Both of those agents are modal agents of rank 0 and so the fact that they defect against CooperateBot imply that FairBot defects against them by theorem 4.1.

• Karl 11 Jun 2013 0:24 UTC

  0 points

  in reply to: Eliezer Yudkowsky’s comment on: Pri­soner’s Dilemma (with visi­ble source code) Tournament

  You’re right, and wubbles’s agent can easily be exploited by a modal agent A defined by A(X)=C <-> [] (X(PB)=C) (where PB is PrudentBot).

• Karl 10 Jun 2013 21:07 UTC

  0 points

  in reply to: Eliezer Yudkowsky’s comment on: Pri­soner’s Dilemma (with visi­ble source code) Tournament

  The agent defined by wubbles is actually the agent called JustBot in the robust cooperation paper and which is proven to be non-exploitable by modal agents.

• Karl 9 Mar 2013 16:11 UTC

  0 points

  in reply to: Wei Dai’s comment on: A prob­lem with “play­ing chicken with the uni­verse” as an ap­proach to UDT

  Well, depend how different the order is...

  If there are theorem in PA of the form “If there are theorems of the form A()≠a and of the form A’()≠a’ then the a and the a’ such that the corresponding theorem come first in the appropriate ordering must be identical.” then you should be okay in the prisoner dilemma setting but otherwise there will be a model of PA in which the two players end up playing different actions and we end up in the same situation as in the post.

  More generally, no matter how you try to cut it, there will always be a model of PA in which all theorems are provable and your agent behavior will always depend on what happen in that model because PA will never be able to prove anything which is false in that model.

• Karl 8 Mar 2013 23:31 UTC

  1 point

  in reply to: Wei Dai’s comment on: A prob­lem with “play­ing chicken with the uni­verse” as an ap­proach to UDT

  If I understand what you’re proposing correctly then that doesn’t work either.

  Suppose it is a theorem of PA that this algorithm return an error, then all theorems of the form A()=a ⇒ U()>=u are demonstrable in PA and so there is no action with the highest u and the algorithm return an error, because that is demonstrable in PA the algorithm must return an error by Löb’s theorem.