gRR

• gRR 18 Feb 2015 6:08 UTC

  5 points

  on: Harry Pot­ter and the Meth­ods of Ra­tion­al­ity dis­cus­sion thread, Fe­bru­ary 2015, chap­ters 105-107

  I am confused about how Philosopher’s stone could help with reviving Hermione. Does QQ mean to permanently transfigure her dead body into a living Hermione? But then, would it not mean that Harry could do it now, albeit temporarily? And, he wouldn’t even need a body. He could then just temporarily transfigure any object into a living Hermione. Also, now that I think of it, he could transfigure himself a Feynman and a couple of Einsteins...

• gRR 10 May 2014 10:53 UTC

  0 points

  in reply to: Houshalter’s comment on: Let’s reim­ple­ment EURISKO!

  The AI can be adapted for other, less restricted, domains

  That the ideas from a safe AI can be used to build an unsafe AI is a general argument against working on (or even talking about) any kind of AI whatsoever.

  The AI adds code that will evolve into another AI into it’s output

  The output is to contain only proofs of theorems. Specifically, a proof (or refutation) of the theorem in the input. The state of the system is to be reset after each run so as to not accumulate information.

  The AI could self-modify incorrectly and result in unfriendly AI

  Any correct or incorrect self-modification is still restricted to the math domain, and so cannot result in an unsafe AI.

  bug in the environment itself

  Guarding against software bugs is easy in this case. You design an abstract virtual machine environment for the AI, then design the software that implements this environment, then formally prove that the software is correct. Guarding against errors caused by cosmic rays is also easy. You estimate the probability of such an error, and then add redundancy until the probability is so low that it won’t happen until the end of the universe.

  Look up how difficult it is to sandbox untrusted code

  Sandboxing untrusted code is easy. The difficult thing is sandboxing it while making it think that it runs normally. This is irrelevant here.

  I don’t believe that a system can work only in formal proofs

  It is quite probable that a “pure math Oracle” system cannot work. The point was, it can be made safe to try.

• gRR 8 May 2014 21:16 UTC

  1 point

  in reply to: Mitchell_Porter’s comment on: [Link] Quan­tum the­ory as the most ro­bust de­scrip­tion of re­pro­ducible experiments

  Well, I liked the paper, but I’m not knowledgeable enough to judge its true merits. It deals heavily with Bayesian-related questions, somewhat in Jayne’s style, so I thought it could be relevant to this forum.

  At least one of the authors is a well-known theoretical physicist with an awe-inspiring Hirsch factor, so presumably the paper would not be trivially worthless. I think it merits a more careful read.

[Link] Quan­tum the­ory as the most ro­bust de­scrip­tion of re­pro­ducible experiments

• gRR 26 Jul 2013 19:58 UTC

  1 point

  on: Harry Pot­ter and the Meth­ods of Ra­tion­al­ity dis­cus­sion thread, part 25, chap­ter 96

  Regarding the “he’s here… he is the end of the world” prophecy, in view of the recent events, it seems like it can become literally true without it being a bad thing. After all, it does not specify a time frame. So Harry may become immortal and then tear apart the very stars in heaven, some time during a long career.

• gRR 26 May 2012 20:25 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  You’re treating resources as one single kind, where really there are many kinds with possible trades between teams

  I think this is reasonably realistic. Let R signify money. Then R can buy other necessary resources.

  But my point was exactly that there would be many teams who could form many different alliances. Assuming only two is unrealistic and just ignores what I was saying.

  We can model N teams by letting them play two-player games in succession. For example, any two teams with nearly matched resources would cooperate with each other, producing a single combined team, etc… This may be an interesting problem to solve, analytically or by computer modeling.

  You still haven’t given good evidence for holding this position regarding the relation between the different Uxxx utilities.

  You’re right. Initially, I thought that the actual values of Uxxx-s will not be important for the decision, as long as their relative preference order is as stated. But this turned out to be incorrect. There are regions of cooperation and defection.

• gRR 26 May 2012 16:53 UTC

  0 points

  in reply to: [deleted]’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  I don’t think you can get an everywhere-positive exchange rate. There are diminishing returns and a threshold, after which, exchanging more resources won’t get you any more time. There’s only 30 hours in a day, after all :)

• gRR 26 May 2012 16:04 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Space (land or whatever is being used). Mass and energy. Natural resources. Computing power. Finite-supply money and luxuries if such exist. Or are you making an assumption that CEVs are automatically more altruistic or nice than non-extrapolated human volitions are?

  These all have property that you only need so much of them. If there is a sufficient amount for everybody, then there is no point in killing in order to get more. I expect CEV-s to not be greedy just for the sake of greed. It’s people’s CEV-s we’re talking about, not paperclip maximizers’.

  Well it does need hardcoding: you need to tell the CEV to exclude people whose EVs are too similar to their current values despite learning contrary facts. Or even all those whose belief-updating process differs too much from perfect Bayesian (and how much is too much?) This is something you’d hardcode in, because you could also write (“hardcode”) a CEV that does include them, allowing them to keep the EVs close to their current values.

  Hmm, we are starting to argue about exact details of extrapolation process...

  There are many possible and plausible outcomes besides “everybody loses”.

  Lets formalize the problem. Let F(R, Ropp) be the probability of a team successfully building a FAI first, given R resources, and having opposition with Ropp resources. Let Uself, Ueverybody, and Uother be the rewards for being first in building FAI, FAI, and FAI, respectively. Naturally, F is monotonically increasing in R and decreasing in Ropp, and Uother < Ueverybody < Uself.

  Assume there are just two teams, with resources R1 and R2, and each can perform one of two actions: “cooperate” or “defect”. Let’s compute the expected utilities for the first team:

  We cooperate, opponent team cooperates:  
     EU("CC") = Ueverybody * F(R1+R2, 0)  
  We cooperate, opponent team defects:  
     EU("CD") = Ueverybody * F(R1, R2) + Uother * F(R2, R1)  
  We defect, opponent team cooperates:  
     EU("DC") = Uself * F(R1, R2) + Ueverybody * F(R2, R1)  
  We defect, opponent team defects:  
     EU("DD") = Uself * F(R1, R2) + Uother * F(R2, R1)
  

  Then, EU(“CD”) < EU(“DD”) < EU(“DC”), which gives us most of the structure of a PD problem. The rest, however, depends on the finer details. Let A = F(R1,R2)/​F(R1+R2,0) and B = F(R2,R1)/​F(R1+R2,0). Then:

  1. If Ueverybody ⇐ Uself*A + Uother*B, then EU(“CC”) < EU(“DD”), and there is no point in cooperating. This is your position: Ueverybody is much less than Uself, or Uother is not much less than Ueverybody, and/​or your team has so much more resources than the other.

  2. If Uself*A + Uother*B < Ueverybody < Uself*A/​(1-B), this is a true Prisoner’s dilemma.

  3. If Ueverybody >= Uself*A/​(1-B), then EU(“CC”) >= EU(“DC”), and “cooperate” is the obviously correct decision. This is my position: Ueverybody is not much less than Uself, and/​or the teams are more evenly matched.

• gRR 26 May 2012 3:15 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  A coalition of 80% of the population forms, which would like to kill the other 20% in order to get their resources

  I have trouble thinking of a resource that would make even one person’s CEV, let alone 80%, want to kill people, in order to just have more of it.

  The question of definition, who is to be included in the CEV? or—who is considered sane?

  This is easy, and does not need any special hardcoding. If someone is so insane that their beliefs are totally closed and impossible to move by knowledge and intelligence, then their CEV is undefined. Thus, they are automatically excluded.

  TDT applies where agents are “similar enough”. I doubt I am similar enough to e.g. the people you labelled insane.

  We are talking about people building FAI-s. Surely they are intelligent enough to notice the symmetry between themselves. If you say that logic and rationality makes you decide to ‘defect’ (=try to build FAI on your own, bomb everyone else), then logic and rationality would make everyone decide to defect. So everybody bombs everybody else, no FAI gets built, everybody loses. Instead you can ‘cooperate’ (=precommit to build FAI<everybody’s CEV> and to bomb everyone that did not make the same precommitment). This gets us a single global alliance.

• gRR 24 May 2012 21:51 UTC

  1 point

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  The resources are not scarce at all. But, there’s no consensus of CEVs. The CEVs of 80% want to kill the rest.

  The resources are not scarce, yet the CEV-s want to kill? Why?

  I meant that the AI that implements your version of CEV would forcibly update people’s actual beliefs to match what it CEV-extrapolated for them.

  It would do so only if everybody’s CEV-s agree that updating these people’s beliefs is a good thing.

  If you believed there were many such people, would you modify your solution, or is ignoring them however many they are fine by you?

  People that would still have false factual beliefs no matter how much evidence and how much intelligence they have? They would be incurably insane. Yes, I would agree to ignore their volition, no matter how many they are.

  The PD reasoning to cooperate only applies in case of iterated PD

  Err. What about arguments of Douglas Hofstadter and EY, and decision theories like TDT?

  Unlike PD, the payoffs are different between players, and players are not sure of each other’s payoffs in each scenario

  This doesn’t really matter for a broad range of possible payoff matrices.

  join research alliance 1, learn its research secrets, then defect and sell the secrets to alliance 2

  Cooperating in this game would mean there is exactly one global research alliance. A cooperating move is a precommitment to abide by its rules. Enforcing such precommitment is a separate problem. Let’s assume it’s solved.

  I’m not convinced by this that it’s an easier problem to solve than that of building AGI or FAI or CEV.

  Maybe you’re right. But IMHO it’s a less interesting problem :)

• gRR 24 May 2012 19:35 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  So you’re OK with the FAI not interfering if they want to kill them for the “right” reasons?

  I wouldn’t like it. But if the alternative is, for example, to have FAI directly enforce the values of the minority on the majority (or vice versa) - the values that would make them kill in order to satisfy/​prevent—then I prefer FAI not interfering.

  “if we kill them, we will benefit by dividing their resources among ourselves”

  If the resources are so scarce that dividing them is so important that even CEV-s agree on the necessity of killing, then again, I prefer humans to decide who gets them.

  So you’re saying your version of CEV will forcibly update everyone’s beliefs

  No. CEV does not updates anyone’s beliefs. It is calculated by extrapolating values in the presence of full knowledge and sufficient intelligence.

  If the original person effectively assigns 0 or 1 “non-updateable probability” to some belief, or honestly doesn’t believe in objective reality, or believes in “subjective truth” of some kind, CEV is not necessarily going to “cure” them of it—especially not by force.

  As I said elsewhere, if a person’s beliefs are THAT incompatible with truth, I’m ok with ignoring their volition. Note, that their CEV is undefined in this case. But I don’t believe there exist such people (excluding totally insane).

  That there exists a possible compromise that is better than total defeat doesn’t mean total victory wouldn’t be much better than any compromise.

  But the total loss would be correspondingly worse. PD reasoning says you should cooperate (assuming cooperation is precommittable).

  If you think so you must have evidence relating to how to actually solve this problem. Otherwise they’d both look equally mysterious. So, what’s your idea?

  Off the top of my head, adoption of total transparency for everybody of all governmental and military matters.

• gRR 24 May 2012 16:25 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  If a majority of humanity wishes to kill a minority, obviously there won’t be a consensus to stop the killing, and AI will not interfere

  The majority may wish to kill the minority for wrong reasons—based on false beliefs or insufficient intelligence. In which case their CEV-s won’t endorse it, and the FAI will interfere. “Fundamentally different” means their killing each other is endorsed by someone’s CEV, not just by themselves.

  But you said it would only do things that are approved by a strong human consensus.

  Strong consensus of their CEV-s.

  Maybe their extrapolated volition simply doesn’t value objective truth highly (because they today don’t believe in the concept of objective truth, or believe that it contradicts everyday experience)

  Extrapolated volition is based on objective truth, by definition.

  If I don’t like the values I might say, thank-you for warning me, now I shall be doubly careful not to evolve into that kind of creature!

  The process of extrapolation takes this into account.

  I think you may be missing a symbol there? If not, I can’t parse it...

  Sorry, bad formatting. I meant four independent clauses: each of the agents does not endorse CEV, but endorses CEV.

  How can a state or military precommit to not having a supersecret project to develop a private AGI?

  That’s a separate problem. I think it is easier to solve than extrapolating volition or building AI.

• gRR 24 May 2012 14:00 UTC

  1 point

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Will it only interfere if a consensus of humanity allows it to do so? Will it not stop a majority from murdering a minority?

  If the majority and the minority are so fundamentally different that their killing each other is not forbidden by the universal human CEV, then no. On what moral grounds would it do the prevention?

  The first AGI that does not favor inaction will become a singleton, destroying the other AIs and preventing future new AIs

  Until everybody agree that this new AGI is not good after all. Then the original AGI will interfere and dismantle the new one (the original is still the first and the strongest).

  you can’t even assume they’ll have a nontrivial CEV at all, let alone that it will “fix” values you happen to disagree with.

  But I can be sure that CEV fixes values that are based on false factual beliefs—this is a part of the definition of CEV.

  I have no idea what your FAI will do

  But you can be sure that it is something about which you (and everybody) would agree, either directly or if you were more intelligent and knew more.

  there are no objectively distinguished morals

  But there may be a partial ordering between morals, such that X<Y iff all “interfering” actions (whatever this means) that are allowed by X are also allowed by Y. Then if A1 and A2 are two agents, we may easily have:

  ~Endorses(A1, CEV) ~Endorses(A2, CEV) Endorses(A1, CEV)
  Endorses(A2, CEV)

  [assuming Endorses(A, X) implies FAI does not perform any non-interfering action disagreeable for A]

  if and when nation-states and militaries realize AGI is a real-world threat, they will go to war with each trying to prevent anyone else from building an AGI first. It’s the ultimate winner-take-all arms race.
  This is going to happen, it might be happening already if enough politicians and generals had the beliefs of Eliezer about AGI, and it will happen (or not) regardless of anyone’s attempts to build any kind of Friendliness theory.

  Well, don’t you think this is just ridiculous? Does it look like the most rational behavior? Wouldn’t it be better for everybody to cooperate in this Prisoner’s Dilemma, and do it with a creditable precommitment?

• gRR 24 May 2012 10:37 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  A FAI that never does anything except prevent existential risk—which, in a narrow interpretation, means it doesn’t stop half of humanity from murdering the other half—isn’t a future worth fighting for IMO. We can do so much better.

  No one said you have to stop with that first FAI. You can try building another. The first FAI won’t oppose it (non-interference). Or, better yet, you can try talking to the other half of the humans.

  There are people who believe religiously that End Times must come

  Yes, but we assume they are factually wrong, and so their CEV would fix this.

  A FAI team that precommitted to implementing CEV would definitely get the most funds. Even a team that precommitted to CEV might get more funds than CEV, because people like myself would reason that the team’s values are closer to my own than humanity’s average, plus they have a better chance of actually agreeing on more things.

  Not bloody likely. I’m going to oppose your team, discourage your funders, and bomb your headquarters—because we have different moral opinions, right here, and if the differences turn out to be fundamental, and you build your FAI, then parts of my value will be forever unfulfilled.

  You, on the other hand, may safely support my team, because you can be sure to like whatever my FAI will do, and regarding the rest, it won’t interfere.

• gRR 23 May 2012 19:01 UTC

  13 points

  on: Prob­le­matic Prob­lems for TDT

  The problems look like a kind of an anti-Prisoner’s Dilemma. An agent plays against an opponent, and gets a reward iff they played differently. Then any agent playing against itself is screwed.

• gRR 22 May 2012 23:52 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  I would be fine with FAI removing existential risks and not doing any other thing until everybody(’s CEV) agrees on it. (I assume here that removing existential risks is one such thing.) And an FAI team that creditably precommitted to implementing CEV instead of CEV would probably get more resources and would finish first.

• gRR 22 May 2012 21:28 UTC

  0 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Well, my own proposed plan is also a contingent modification. The strongest possible claim of CEV can be said to be:

  There is a unique X, such that for all living people P, CEV

  = X.

  Assuming there is no such X, there could still be a plausible claim:

  Y is not empty, where Y = Intersection{over all living people P} of CEV

  .

  And then AI would do well if it optimizes for Y while interfering the least with other things (whatever this means). This way, whatever “evolving” will happen due to AI’s influence is at least agreed upon by everyone(’s CEV).

• gRR 22 May 2012 20:54 UTC

  2 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Back here you said “Well, perhaps yes.” I understand that to mean you agree with my point that it’s wrong /​ bad for the AI to promote extrapolated values while the actual values are different and conflicting

  I meant that “it’s wrong/​bad for the AI to promote extrapolated values while the actual values are different and conflicting” will probably be a part of the extrapolated values, and the AI would act accordingly, if it can.

  My position is that the AI must be guided by the humans’ actual present values in choosing to steer human (social) evolution towards or away from possible future values. This has lots of downsides, but what better option is there?

  The problem with the actual present values (beside the fact that we cannot define them yet, no more than we can define their CEV) is that they are certain to not be universal. We can be pretty sure that someone can be found to disagree with any particular proposition. Whereas, for CEV, we can at least hope that a unique reflectively-consistent set of values exists. If it does and we succeed to define it, then we’re home and dry. Meanwhile, we can think of contingency plans about what to do if it does not or we don’t, but the uncertainty about whether the goal is achievable does not mean that the goal itself is wrong.

• gRR 22 May 2012 16:50 UTC

  4 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  Humans don’t know which of their values are terminal and which are instrumental, and whether this question even makes sense in general. Their values were created by two separate evolutionary processes. In the boxes example, humans may not know about the diamond. Maybe they value blue boxes because their ancestors could always bring a blue box to a jeweler and exchange it for food, or something.

  This is precisely the point of extrapolation—to untangle the values from each other and build a coherent system, if possible.

• gRR 22 May 2012 16:20 UTC

  2 points

  in reply to: DanArmak’s comment on: Holden’s Ob­jec­tion 1: Friendli­ness is dangerous

  No, the “actual” values would tell it to give the humans the blue boxes they want, already.