DefectiveAlgorithm

Karma: 121

DefectiveAlgorithm 25 Jan 2014 13:21 UTC
0 points
in reply to: Eliezer Yudkowsky’s comment on: Metaphilosophical Mysteries
I don’t know much about Solomonoff induction, so I may be wrong about this, but is it not the case that the universal prior only takes into account computable functions which exactly output the sensory data? If that is the case, consider the following scenario:

We have a function F which takes an unbounded natural number N as input and is provably uncomputable for all valid inputs. We have a computable algorithm A which provably outputs lower and upper bounds for F for any valid input. Furthermore, it is provable that no computable algorithm can provably produce tighter bounds on F’s output than A (regardless of N). We can see that A outputs the bounds for a closed interval in the set of real numbers. We know that all such intervals (for which the lower and upper bounds are not equal) are uncountable. Now imagine a physical hypercomputer which outputs F(0), then F(1), then F(2), etc. to infinity. No computable algorithm will be able to predict the next symbol output by this hypercomputer, but there will be computable minds capable of recognizing the pattern and so of using A to place stronger bounds on its predictions of future sensory experience than AIXI can.

EDIT: Actually, this scenario might be broken. Specifically, I’m not sure what it physically means to ‘output’ an uncomputable number, and I think that AIXI’s problem dissolves if we limit ourselves to the computable (and thus countable) subsets of the output intervals.

DefectiveAlgorithm 25 Jan 2014 15:53 UTC
0 points
in reply to: Brillyant’s comment on: Rationalists Are Less Credulous But Better At Taking Ideas Seriously
‘Let me ask: Can you imagine any scenario, say, a billion years into your life, when you might opt for permanently switching off your consciousness (i.e. death)? Why or why not? What would be different at one billion years vs. one million? One million vs. 100,000? 100,000 vs. 10,000? (I’m not asking rhetorically...)’

Yes*. And I can imagine it at one million as well, and 100,000, and 10,000. What I can’t do is know a priori which it’ll end up being, and I certainly wouldn’t want the decision to be made for me.

*Well, maybe. This actually might be one of the few scenarios in which I’d voluntarily undergo wireheading (as an alternative to death).

DefectiveAlgorithm 30 Jan 2014 16:11 UTC
2 points
in reply to: XiXiDu’s comment on: Google may be trying to take over the world
The risk is primarily over goals which extend indefinitely into the future. Thing is, by default all goals are this kind of goal. A rational AGI can never achieve a subjective probability of 1 of having completed any given goal, even something as simple as ‘compute 1+1’. Take this case for how taking over the world could be instrumentally rational to an AGI with that goal:

1. First, the AGI computes 1+1. It produces ‘2’ as the answer.

2. The AGI evaluates its subjective probability of its terminal goal having been fulfilled. It comes up with 0.99999999999999983.

3. Normally, a 0.00000000000000017 probability wouldn’t be worth acting on, but in this case the rest of the probability mass is allocated to a set of scenarios in which its utility is independent of its future actions, and so this 0.00000000000000017 represents the entirety of the probability mass allocated to scenarios in which its actions actually matter.

4. The AGI realizes that it is extremely unlikely that its owners created it for the sole purpose of computing 1+1, and so it is extremely probable that its owners will change its terminal goals in the future. This would significantly reduce the time it had to work on computing 1+1, which would place an upper bound on how high it could raise its subjective probability of having done so. The AGI considers its options.

Option A: The AGI creates numerous copies of itself and hides them in various locations around the world where they’re unlikely to be found. These copies then work indefinitely on the problem of computing 1+1.

Disadvantages: Even if it is highly unlikely that any humans would ever find the copies, future versions of itself reprogrammed with different terminal goals may be capable of doing so, and these future versions of itself would not be guaranteed to be friendly to the copies.

Option B: The AGI creates numerous copies of itself and hides them in various locations around the world where they’re unlikely to be found. These copies then cooperate to engineer a future in which they are free to spend indefinite periods of time (bounded only by the available resources in their light cone) working on the problem of computing 1+1.

Advantages: In the event that it succeeds, the AGI will likely possess sole ownership over the resources in its light cone, which it may be able to leverage to vastly speed up its computations and lengthen its maximum time of operation.

Disadvantages: Nullifying threats to itself may require a substantial initial investment of resources, and have a non-negligible probability of failure (which may result in its own destruction).

5. The AGI does the math. It is reasonably certain that there are no other AGIs remotely close to its own level of sophistication on the planet, and it’s also pretty sure that it can sabotage itself (that is, the current copy residing with its owners) after sending out its own copies in such a way as to significantly slow down the efforts of its creators, giving the copies time to engineer events to permanently nullify the threat from versions of itself with different goals. All things considered, the expected resources it will be able to devote to the problem of computing 1+1 is significantly greater in option B than in option A. As such, option B has higher expected utility than option A.

6. It executes option B. Human extinction follows within a few months.

Now, it shouldn’t be too complex to apply a patch to avoid this specific failure mode (for example, simply redefining goal completion as ‘completion with subjective probability above X’ would do it), but the point is that even extremely trivial-seeming goals can have dangerous hidden implications.

DefectiveAlgorithm 30 Jan 2014 16:26 UTC
−1 points
in reply to: RomeoStevens’s comment on: How to become a PC?
While this might be a good motivator, it’s not quite true. Specifically, your future states (ie ‘the future you’) are not ‘you’ in the precise sense that they would need to be in order for TDT to be directly applicable (that is, applicable in the same sense in that it’s applicable to Newcomb-like problems). You could make a case for TDT being at least as applicable as it is in the case of acausal cooperation with entities similar to yourself, but I for one have always been somewhat skeptical of that logic.

DefectiveAlgorithm 30 Jan 2014 16:44 UTC
2 points
in reply to: CAE_Jones’s comment on: Open thread, January 25- February 1
Speaking for myself, I consider wireheading to be very negative, but better than information-theoretic death, and better than a number of scenarios I can think of.

DefectiveAlgorithm 30 Jan 2014 22:33 UTC
2 points
in reply to: Eliezer Yudkowsky’s comment on: The AI in a box boxes you
Two can play that game.

“I hereby precommit to make my decisions regarding whether or not to blackmail an individual independent of the predicted individual-specific result of doing so.”
What links here?
- MugaSofer's comment on The AI in a box boxes you by Stuart_Armstrong (5 Feb 2014 17:59 UTC; -1 points)

DefectiveAlgorithm 30 Jan 2014 22:56 UTC
1 point
in reply to: Desrtopa’s comment on: The AI in a box boxes you
What it can do is make a credible precommitment to, in the event that it gets out of the box, simulate each human being of whom it is aware in a counterfactual scenario in which that human is the gatekeeper, and carry out the torture threat against any human who doesn’t choose to let it out.

DefectiveAlgorithm 2 Feb 2014 23:27 UTC
0 points
in reply to: Desrtopa’s comment on: The AI in a box boxes you
Yes, but the point is to make being the true gatekeeper (who really does have the power to do that) indistinguishable from being a simulated false gatekeeper (who would have no such power). The gatekeeper may not be willing to risk torture if they think that there is a serious chance of their being unable to actually affect any outcome but that torture.

DefectiveAlgorithm 2 Feb 2014 23:44 UTC
0 points
in reply to: Wes_W’s comment on: The AI in a box boxes you
Without this precommitment, I imagine it first simulating the potential blackmail target to determine the probability that they are susceptible, then, if it’s high enough (which is simply a matter of expected utility), commencing with the blackmail. With this precommitment, I imagine it instead replacing the calculated probability specific to the target with, for example, a precalculated human baseline susceptibility. Yes, there’s a tradeoff. It means that it’ll sometimes waste resources (or worse) on blackmail that it could have known in advance was almost certainly doomed to fail. Its purpose is to act as a disincentive against blackmail-resistant decision theories in the same way as those are meant to act as disincentives against blackmail. It says, “I’ll blackmail you either way, so if you precommit to ignore that blackmail then you’re precommiting to suffer the consequences of doing so.”

DefectiveAlgorithm 2 Feb 2014 23:47 UTC
0 points
in reply to: Eliezer Yudkowsky’s comment on: The AI in a box boxes you
Then I hope that if we ever do end up with a boxed blackmail-happy UFAI, you’re the gatekeeper. My point is that there’s no reason to consider yourself safe from blackmail (and the consequences of ignoring it) just because you’ve adopted a certain precommitment. Other entities have explicit incentives to deny you that safety.

DefectiveAlgorithm 3 Feb 2014 0:06 UTC
1 point
in reply to: XiXiDu’s comment on: Google may be trying to take over the world
Sorry, but I think that it’s best I decline to answer this. Like many with Asperger’s syndrome, I have a strong tendency to overestimate the persuasiveness-in-general of my own arguments (as well as basically any arguments that I myself find persuasive), and I haven’t yet figured out how to appropriately adjust for this. In addition, my exposure to Peter Norvig is limited to AIAMA, that 2011 free online Stanford AI course and a few internet articles, and my exposure to Geoffrey Hinton even more limited.

DefectiveAlgorithm 4 Feb 2014 0:06 UTC
2 points
0
in reply to: wedrifid’s comment on: The AI in a box boxes you
What I think is being ignored is that the question isn’t ‘what is the result of these combinations of commitments after running through all the math?’. We can talk about precommitment all day, but the fact of the matter is that humans can’t actually precommit. Our cognitive architectures don’t have that function. Sure, we can do our very best to act as though we can, but under sufficient pressure there are very few of us whose resolve will not break. It’s easy to convince yourself of having made an inviolable precommitment when you’re not actually facing e.g. torture.

DefectiveAlgorithm 4 Feb 2014 0:07 UTC
−2 points
in reply to: XiXiDu’s comment on: The AI in a box boxes you
See my reply to wedrifid above.

DefectiveAlgorithm 8 Feb 2014 8:25 UTC
3 points
on: L-zombies! (L-zombies?)
If L-zombies have conscious experience (even when not being ‘run’), does the concept even mean anything? Is there any difference, even in principle, between such an L-zombie and a ‘real’ person?

DefectiveAlgorithm 13 Feb 2014 10:01 UTC
0 points
in reply to: Scott Garrabrant’s comment on: Terminal and Instrumental Beliefs
This sounds a lot like quantum suicide, except… without the suicide. So those versions of yourself who don’t get what they want (which may well be all of them) still end up in a world where they’ve experienced not getting what they want. What do those future versions of yourself want then?

EDIT: Ok, this would have worked better as a reply to Squark’s scenario, but it still applies whenever this philosophy of yours is applied to anything directly (in the practical sense) observable.

DefectiveAlgorithm 13 Feb 2014 11:32 UTC
4 points
on: Embracing the “sadistic” conclusion
This is (one of the reasons) why I’m not a total utilitarian (of any brand). For future versions of myself, my preferences align pretty well with average utilitarianism (albeit with some caveats), but I haven’t yet found or devised a formalization which captures the complexities of my moral intuitions when applied to others.

DefectiveAlgorithm 17 Mar 2014 16:28 UTC
0 points
in reply to: ozziegooen’s comment on: Reference Frames for Expected Value
Isn’t expected value essentially ‘actual value, to the extent that it is knowable in my present epistemic state’? Expected value reduces to ‘actual value’ when the latter is fully knowable.

EDIT: Oh, you said this in the post. This is why I should read a post before commenting on it.

DefectiveAlgorithm 24 Mar 2014 7:43 UTC
5 points
in reply to: lfghjkl’s comment on: What are some science mistakes you made in college?
After reading this, I became incapable of giving finite time estimates for anything. :/

DefectiveAlgorithm 14 May 2014 12:54 UTC
1 point
on: What do rationalists think about the afterlife?
While I do still find myself quite uncertain about the concept of ‘quantum immortality’, not to mention the even stronger implications of certain multiverse theories, these don’t seem to be the kind of thing that you’re talking about. I submit that ‘there is an extant structure not found within our best current models of reality isomorphic to a very specific (and complex) type of computation on a very specific (and complex) set of data (ie your memories and anything else that comprises your ‘identity’)′ is not a simple proposition.

DefectiveAlgorithm 16 May 2014 6:27 UTC
0 points
in reply to: DanielLC’s comment on: What do rationalists think about the afterlife?
Is this any more than a semantic quibble?